Abstract
Background
Two systems measure surgical site infection rates following colorectal surgeries. Center for Medicare and Medicaid Services pay-for-performance initiatives use National Healthcare Safety Network data for hospital comparisons.
Objective
Compare database concordance.
Design
Multi-institution cohort study of system-wide Colorectal Surgery Collaborative. The National Surgical Quality Improvement Program requires rigorous, standardized data capture techniques; National Healthcare Safety Network allows five data capture techniques. Standardized surgical site infection rates were compared between databases. Cohen's Kappa coefficient calculated.
Setting
Boston-area hospitals.
Patients
National Healthcare Safety Network or National Surgical Quality Improvement Program patients undergoing colorectal surgery.
Main Outcome Measures
Standardized surgical-site infection rates
Results
Thirty-day surgical-site infection rates of 3,547 (National Surgical Quality Improvement Program) vs 5,179 (National Healthcare Safety Network) colorectal procedures (2012-2014). Discrepancies appeared: National Surgical Quality Improvement Program database of Hospital 1 (N=1,480 patients) routinely found surgical-site infection rates around 10%, routinely deemed rate “exemplary” or “as expected” (100%). National Healthcare Safety Network data from the same hospital and time period (N=1,881) revealed similar overall surgical-site infection rate (10%), but standardized rates were deemed “worse than national average” 80% of the time. Overall, hospitals using less rigorous capture methods had improved surgical-site infection rates for National Healthcare Safety Network compared to standardized National Surgical Quality Improvement Program reports. The correlation coefficient between standardized infection rates was 0.03 (p=0.88). During 25 site-time period observations, National Surgical Quality Improvement Program and National Healthcare Safety Network data matched for 52% of observations (13/25). Kappa=0.10 (95% CI: -0.1366-0.3402; p=0.403), indicating poor agreement.
Limitations
Hospitals located in Northeastern United States only.
Conclusions
Variation in Center for Medicare and Medicaid Services-mandated National Healthcare Safety Network infection surveillance methodology leads to unreliable results, which is apparent when these results are compared to standardized data. High quality data would improve care quality and compare outcomes amongst institutions.
Keywords: colon surgery, surgical site infection, infection rate, surgical quality, Center for Medicare & Medicaid Services
Introduction
Surgeons share a strong interest in improving the quality of patient care and reducing rates of surgical site infection (SSI). SSIs have previously been found to be associated with increased morbidity, length of hospital stay, and overall cost.1-4 Thus, SSI rate is a common metric used in surgical outcomes research to evaluate surgical quality. However, SSI rate after colon surgery is influenced by several other factors, including patient comorbid diseases, presentation and type of illness, surgical approach, and case complexity.5-8 If SSI rate is to be used accurately to assess the quality of care provided by a surgeon or a hospital for colectomy, detailed data on these potential confounding factors must be accurately collected and the evaluator must take that data into account.
Recently, the Center for Medicare & Medicaid Services (CMS) pay-for-performance initiatives mandated that hospitals enter data into the Center for Disease Control (CDC) National Healthcare Safety Network (NHSN) database.9, 10 The data collected in this database was originally intended for identification of broad epidemiologic trends in SSIs across the country. Because the goal was to only assess large SSI trends.11, 12 the data collection methods were not well standardized. Previous studies have found large discrepancies in the SSI rates reported in NHSN data compared to internal audits,13-15 and their utilization of International Classification of Diseases, Ninth Revision (ICD-9) codes to identify surgical cases for inclusion has been found inaccurate due to non-physician coding.16, 17 Nonetheless, this data is now being used to penalize hospitals: if CMS, reviewing NHSN data, deems infection rates “too high,” these hospitals may lose a percentage of their entire CMS hospital revenue.
In parallel, SSI data is collected at many hospitals through the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP). The goal of this surgeon-led program is to track surgical complications, including SSIs, on an apples-to-apples basis, so as to guide quality improvement initiatives at these institutions.18
Given the uncertainty as to whether current NHSN guidelines ensure quality data collection, the aim of this study is to 1) assess the current state of data reporting to NHSN databases across a healthcare network system and 2) determine the concordance of NHSN data with the more rigorous, highly standardized NSQIP database. We hypothesize a low level of concordance of standardized surgical site infection rates between these databases; moreover, we think the NHSN system will reveal lower SSI rates in smaller community hospitals compared to large, academic hospitals due to the lack of case adjustment.
Methods
Cohort Description
This is a multi-institution cohort study comparing SSI data collected through two national databases used to document and report 30-day postoperative SSI rates following colorectal operations: the ACS NSQIP and CDC NHSN databases. Data following colon surgery performed at five hospitals participating in our system wide Partners Colorectal Collaborative between 2012 and 2014 was gathered. This collaborative includes two high volume academic hospitals and three lower volume community hospitals (performing 50 versus 0-15 major colorectal procedures per month, respectively). The hospitals are labeled Hospital 1-5 based on case volume.
The NSQIP Database
All hospitals attempt to collect 100% of colectomy outcomes data for the NSQIP, and colectomy cases are identified by CPT codes. Certain colectomies are excluded in the 100% capture method: cases performed during nursing reviewer vacations (Hospitals 2-5), colectomies performed by gynecologic providers in the year 2012 (Hospital 1-5), colectomies performed due to trauma, due to a complication from a primary surgery (e.g. colectomy with heart surgery within 30 days), or those performed concurrently with another major case (e.g. hepatectomy with colectomy). Thus, approximately 85% of colonic resections are included in the database.
ASCS NSQIP outcomes data is collected prospectively, through a nationally standardized protocol. Trained surgical clinical nurse reviewers query the medical record and monitor patients for 30-days post-discharge through patient phone calls and direct discussions with visiting nurses, outside emergency rooms and offices of primary care. All readmissions and medical care received post-discharge at any facility is captured to document 30-day complications.
The NSQIP database risk adjustment model includes preoperative, intraoperative, and postoperative variables including CPT codes, information about patient complexity, diagnoses, and comorbidities, which are obtained using standardized and concrete data collection methods. The model is risk-adjusted and case mix-adjusted to account for potential confounders.18, 19 Observed/Expected (O/E) SSI Rate Ratios are calculated using a hierarchical multivariable logistic regression model (Bayesian shrinkage or reliability adjustment).
The NHSN Database
All hospitals have a separate team under the Infectious Disease Department's supervision to collect 30-day postoperative SSI data for the CDC NHSN database, which uses medical ICD-9 billing codes to identify the cases for its denominator. The aim for this database is to include 100% of all cases involving any form of surgical colonic manipulation (both resection and repair). The data is entered into the NHSN database by a staff member with infection prevention experience, such as an RN or MD. In contrast to NSQIP, NHSN does not exclude pediatric patients, operations for trauma, re-operation to address preexisting colonic infection, performance of concurrent cases and/or multi visceral resections, or surgeries performed to address a complication of another surgical procedure.
Data collection for NHSN SSIs is not standardized from hospital to hospital and can be collected using a number of CDC approved techniques: 1) direct examination of patients' wounds during follow-up visits to either surgery clinics or physicians' offices, 2) review of medical records or surgery clinic patient records, 3) surgeon surveys by mail or telephone, 4) patient surveys by mail or telephone with omissions being marked as no infection, or 5) any combination of the above.20 Table 1 describes the method of infection data collection employed by each hospital in the cohort. Hospital 1 was the only hospital to continue rigorous data capture methods similar to the NSQIP.
Table 1.
Data collection methods used by each hospital. Highlighted hospitals have consistent methods of data collection across databases.
| Hospital | ACS NSQIP data collection Method | NHSN data collection method |
|---|---|---|
| 1 | 100% review with phone calls, and chart review | Review of 100% of NHSN qualifying cases through medical record chart review, phone calls and wound culture queries |
| 2 | 100% review with phone calls, and chart review | Hospital readmission and wound culture trigger used for subsequent chart review of hospital records on the patients identified as high risk for SSI |
| 3 | 100% review with phone calls, and chart review | Hospital readmission and wound culture trigger used for subsequent chart review of hospital records on the patients identified as high risk for SSI |
| 4 | 100% review with phone calls, and chart review | Hospital readmission and wound culture trigger used for subsequent chart review of hospital records on the patients identified as high risk for SSI |
| 5 | 100% review with phone calls, and chart review | Surgeon self-reported surveys |
The NHSN calculates a standardized infection ratio (SIR) utilizing multivariable logistic regression. Collected infection rates are compared to a previous year's baseline period of data as a benchmark. Expected infection rates are adjusted for age, anesthesia, ASA class, duration of surgery, medical school affiliation, bed size, utilization of laparoscopy and wound classification. Variables such as diabetes mellitus, obesity, immunosuppression, cancer, malnutrition, chemotherapy, or other operations/case-mix are not taken into account in the adjustment.
Database Comparison
IRB and Partners Colorectal Collaborative approval was obtained. A step-wise data gathering process was used to compare the two databases at the hospital level. Rates of SSI were calculated as number of infection over number of cases identified as colectomy by the database. NSQIP data was used to describe institutional case mix. Cases were matched between the databases using unique patient identifiers and date of surgery. The NHSN cases not identified in the NSQIP database were matched back to their respective CPT codes and/or patient operative notes to obtain granular detail about these operations.
Statistical Analysis
Data gathered by NSQIP and NHSN databases was compared. Descriptive statistics were reported as percentages for categorical variables. Univariate analysis using General Chi Square test was performed to determine SSI differences and case mix differences between hospitals. Spearman correlation between NSQIP and NHSN SSI rates was assessed by comparing O/E ratios reported by the NSQIP database and SIR reported by the NHSN database. A Cohen's Kappa coefficient was obtained to compare reliability of these two SSI endpoints. Analysis was performed using STATA version 12 (College Station, TX: StataCorp LP).
Results
Hospitals 1-5 reported their 2012-2014 SSI rates following 3,547 colectomies identified by NSQIP and 5,179 colon procedures identified by NHSN, see Figure 1 for cohort description. A careful review of the case mix determined by the CPT codes included in the NSQIP database showed that Hospitals 1-5 performed cases of different complexities, with the overall distribution of right, left, total and other colectomies being significantly different across hospitals, see Table 2. In general, academic hospitals (which were the two highest volume hospitals) performed cases of higher complexity. For example, the higher volume hospitals (Hospital 1 and 2) performed more total colectomies at a rate of 7% compared to 1-2% at smaller volume hospitals (p<0.001). The use of laparoscopy was center dependent ranging from 13% to 61% of cases (p<0.001). In addition, the rate of colostomy/end ileostomy utilization varied by hospital with a range of 12% to 21% of cases (p<0.001). The complexity of cases of each of the participating hospitals is depicted in Figure 2.
Figure 1. Flowchart of cohort.
Table 2. Description of case mix by hospital.
| Overall | Hospital 1 | Hospital 2 | Hospital 3 | Hospital 4 | Hospital 5 | p-value | |
|---|---|---|---|---|---|---|---|
| Total NSQIP cases | 3547 | 1480 (41.7%) | 1293 (36.5%) | 365 (10.3%) | 347(9.8%) | 62 (1.7%) | <0.001 |
| Total NHSN cases | 5179 | 1881 (36.3%) | 2044 (39.5%) | 416 (8.0%) | 638 (12.3%) | 200 (3.9%) | |
| Surgical Approach | <0.001 | ||||||
| Open cases | 1987 (56.0%) | 730 (49.3%) | 837 (64.7%) | 229 (62.7%) | 137 (39.5%) | 54 (87.1%) | |
| Laparoscopic cases | 1560 (44.0%) | 750 (50.7%) | 456 (35.3%) | 136 (37.3%) | 210 (60.5%) | 8 (12.9%) | |
| Extent/Location of Resection | <0.001 | ||||||
| Total Colectomy | 205 (5.8%) | 103 (7.0%) | 91 (7.0%) | 5 (1.4%) | 5 (1.4%) | 1 (1.6%) | |
| Right Colectomy | 1036 (29.2%) | 534 (36.1%) | 358 (27.7%) | 129 (35.3%) | 114 (32.9%) | 15 (24.2%) | |
| Left (Colectomy with Low Anastomosis) | 1316 (37.1%) | 580 (39.2%) | 518 (40.1%) | 104 (28.5%) | 105 (30.3%) | 9 (14.5%) | |
| Other Colectomy | 990 (27.9%) | 263 (17.8%) | 326 (25.2%) | 127 (34.8%) | 123 (35.4%) | 37 (59.7%) | |
| Colostomy / Ileostomy Utilization | 616 (17.4%) | 237 (16.0%) | 270 (20.9%) | 56 (15.3%) | 42 (12.1%) | 11 (17.7%) | <0.001 |
| Type of Operation by Complexity | |||||||
| Laparoscopic right colectomy | 564 (15.9%) | 266 (18.0%) | 159 (12.3%) | 58 (15.9%) | 80 (23.1%) | 1 (1.6%) | |
| Open right colectomy | 586 (16.5%) | 268 (18.1%) | 199 (15.4%) | 71 (19.5%) | 34 (9.8%) | 14 (22.6%) | |
| Laparoscopic partial colectomy | 235 (6.6%) | 83 (5.6%) | 64 (4.9%) | 27 (7.4%) | 56 (16.1%) | 5 (8.1%) | |
| Open partial colectomy | 346 (9.8%) | 103 (7.0%) | 123 (9.5%) | 61 (16.7%) | 34 (9.8%) | 25 (40.3%) | |
| Laparoscopic left colectomy | 683 (19.3%) | 368 (24.9%) | 191 (14.8%) | 51 (14.0%) | 72 (20.7%) | 1 (1.6%) | |
| Open left colectomy | 928 (26.2%) | 289 (19.5%) | 466 (36.0%) | 92 (25.2%) | 66 (19.0%) | 15 (24.2%) | |
| Laparoscopic total colectomy | 78 (2.2%) | 33 (2.2%) | 42 (3.2%) | 0 (0.0%) | 2 (0.6%) | 1 (1.6%) | |
| Open total colectomy | 127 (3.6%) | 70 (4.7%) | 49 (3.8%) | 5 (1.4%) | 3 (0.9%) | 0 (0.0%) |
Figure 2. Case mix complexity by hospital.
More granular data on the types of operations included in the NHSN colonic database and not the NSQIP colectomy database, was obtained. For hospital 1, only 1554 of 1881 cases included in the NHSN denominator (82.6%) were found to be colectomies. 171 colectomy cases that were identified by the NSQIP reviewers were erroneously omitted from the NHSN denominator due to inaccurate ICD-9 coding. NHSN reviewers also incorrectly included 327 cases involving operations miscoded as a colonic case by hospital billers. This included 161 miscellaneous bowel operations (not colectomy or proctectomy by CPT code), 103 proctectomies (which should not have been included due to higher expected complication rates), and 30 procedures involving the small intestine only. Other procedures that were included as colon operations under NHSN at <10 cases over the study period were flap operations, upper gastrointestinal operations, hepatectomy, pancreatectomy, and abdominal explorations. Similar findings were obtained on review of hospital 2; see Table 3 for further details. A few of these additional cases had unusually high SSIs (SSI in 40-100% of cases). 572 NHSN colectomies were not in the NSQIP database and the surgical indication for these procedures was assessed. In Hospital 1, some of these surgical indications included: ischemic bowel following complications caused by major cardiac/vascular surgery, a redo colectomy with end stoma performed 9 days after a patient had a colectomy and an anastomotic leak at an outside hospital, a case using a colonic conduit for an emergent esophageal reconstruction on a patient that ingested lye, various colon repairs following stab wounds or gunshot wounds to the abdomen, and various complex Hartman closure procedures. All of these cases were correctly excluded from the NSQIP database.
Table 3. List of cases added to Hospital 1 NHSN denominator CPT, name, SSI rate.
| Hospital 1 | Hospital 2 | |||||
|---|---|---|---|---|---|---|
| NHSN Cases from Hospital 1- CPT description | # of cases | # of SSI | SSI Rate | # of cases | # of SSI | SSI Rate |
| Colectomy CPT | 1554 | 148 | 9.5% | 1228 | 122 | 9.9% |
| Misc Bowel surgery - not colectomy or proctectomy CPT | 161 | 22 | 13.7% | 7 | 1 | 14.3% |
| Proctectomy | 103 | 10 | 9.7% | 128 | 5 | 3.9% |
| Small Intestine | 30 | 7 | 23.3% | 246 | 18 | 7.3% |
| Flap | 5 | 2 | 40.0% | 6 | 1 | 16.7% |
| Misc UGI/Stomach/Esophagus | 5 | 0 | 0.0% | 27 | 1 | 3.7% |
| Uro/GYN | 5 | 2 | 40.0% | 22 | 2 | 9.1% |
| Appendectomy | 4 | 0 | 0.0% | 3 | 0 | 0% |
| Misc Rectal | 4 | 0 | 0.0% | 14 | 0 | 0% |
| Hepatectomy | 2 | 0 | 0.0% | 15 | 2 | 13.3% |
| Pancreatectomy | 2 | 0 | 0.0% | 16 | 1 | 6.3% |
| Unlisted Proc Abdomen | 2 | 0 | 0.0% | 86 | 8 | 9.3% |
| Explore Abdomen | 1 | 0 | 0.0% | 3 | 0 | 0% |
| Hernia Inguinal | 1 | 0 | 0.0% | 1 | 1 | 100% |
| Misc - mouth/throat/esophagus proc. | 1 | 1 | 100.0% | 2 | 0 | 0% |
| Windpipe/Airway Procedures | 1 | 1 | 100.0% | 0 | 0 | 0% |
| Colostomy only | 0 | 0 | 0% | 142 | 1 | 0.7% |
| Unable to obtain details | 0 | 0 | 0% | 98 | 3 | 3.1% |
| Grand Total | 1881 | 193 | 10.3% | 2044 | 166 | 12.3% |
| NHSN COLO Cases Matched to NSQIP Colectomy Cases | n | % of NHSN Cases | % of NSQIP Cases | n | % of NHSN Cases | % of NSQIP Cases |
| Matched Denominator Cases - NSQIP and NHSN | 1309 | 69.6% | 88.4% | |||
| Didn't Match - were in NHSN COLO and NOT NSQIP Colectomy | 572 | 30.4% | ||||
| Didn't Match that had a Colectomy CPT billed | 251 | |||||
| Didn't Match - were in NSQIP Colectomy and NOT NHSN COLO | 171 | 11.6% | ||||
| NHSN Infection Cases Matched to NSQIP | n | % of NHSN Infection Cases | % of NSQIP Infection Cases | n | % of NHSN Cases | % of NSQIP Cases |
| Matched Numerator Cases - NSQIP and NHSN - infections | 80 | 41.5% | 67.8% | |||
| Unmatched to any NSQIP case | 78 | 40.4% | ||||
| Unmatched to NSQIP w/ no Colectomy CPT | 43 | 22.3% | ||||
| Matched to NSQIP case but NOT infection in NSQIP | 35 | 18.1% | ||||
| NSQIP SSI not matched to NHSN Infection List | 38 | 32.2% | ||||
The overall number of infections between NSQIP and NHSN was different across all hospitals with infection rates of 12.3% and 8.2%, respectively. Table 4 describes the number of superficial, deep and organ space infections at each hospital as well as the overall number of infections by procedure type. Overall, open colectomy operations had higher infection rates than laparoscopic operations with the highest infection rate occurring in open partial colectomy cases (56.4%). The overall infection rate between NSQIP and NHSN did not match for any of the hospitals with no clear trend in reporting differences. This random variability in infection rate reported was sometimes large (e.g. 3.3% versus 10.7% in Hospital 3). The difference in infection rate between the databases was due to discrepancies in both the denominator (number of colon cases identified) and the numerator (number of infections identified). Table 5 shows that NHSN reports a higher number of colon cases at every hospital with a 40% increase in number of colon cases reported on average. Academic hospitals (1,2) increased the N in their denominator by 27% and 58%, respectively, while community hospitals (3,4,5) increased their N by 52%, 14% and 58%, respectively. Despite higher denominators, the overall number of infections detected by NHSN during the same time interval was lower for every hospital except hospital 1. This resulted in an increase in infection rate in NHSN versus NSQIP for Hospital 1 by 2.3% and a decrease in infection rate for all other hospitals, -9.3%, -6.7%, -7.3%, -1.4%.
Table 4. Comparison of surgical site infections by hospital and CPT code.
| Overall | Hospital 1 | Hospital 2 | Hospital 3 | Hospital 4 | Hospital 5 | p-value | |
|---|---|---|---|---|---|---|---|
| Superficial Infection | 275 (7.8%) | 60 (4.1%) | 143 (11.1%) | 26 (7.5%) | 46 (12.6%) | 0 (0.0%) | <0.001 |
| Deep Infection | 18 (0.5%) | 4 (0.3%) | 11 (0.9%) | 1 (0.3%) | 1 (0.3%) | 1 (1.6%) | <0.001 |
| Organ Space Infection | 144 (4.1%) | 61 (4.1%) | 58 (4.5%) | 10 (2.9%) | 15 (4.1%) | 0 (0.0%) | <0.001 |
| Laparoscopic right colectomy | 42 (7.4%) | 18 (6.8%) | 18 (11.3%) | 6 (10.3%) | 0 (0.0%) | 0 (0.0%) | |
| Open right colectomy | 91 (15.5%) | 31 (11.6%) | 43 (21.6%) | 16 (22.5%) | 0 (0.0%) | 1 (7.1%) | |
| Laparoscopic partial colectomy | 20 (8.5%) | 4 (4.8%) | 3 (4.7%) | 2 (7.4%) | 11 (19.6%) | 0 (0.0%) | |
| Open partial colectomy | 195 (56.4%) | 34 (33.0%) | 116 (94.3%) | 25 (41.0%) | 20 (58.8%) | 0(0.0%) | |
| Laparoscopic left colectomy | 51 (7.5%) | 23 (6.3%) | 14 (7.4%) | 11 (21.6%) | 3 (4.2%) | 0 (0.0%) | |
| Open left colectomy | 11 (1.2%) | 4 (1.4%) | 4 (0.9%) | 1 (1.1%) | 2 (3.0%) | 0 (0.0%) | |
| Laparoscopic total colectomy | 7 (9.0%) | 2 (6.1%) | 5 (11.9%) | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) | |
| Open total colectomy | 20 (15.7%) | 9 (12.9%) | 9 (18.4%) | 1 (20.0%) | 1 (33.3%) | 0 (0.0%) | |
| Total NSQIP infections | 437 (12.3%) | 125 (8.4%) | 212 (16.4%) | 37 (10.7%) | 62 (17.0%) | 1 (1.6%) | <0.001 |
| Total NHSN infections | 423 (8.2%) | 193 (10.3%) | 166 (8.1%) | 21 (3.3%) | 38 (9.1%) | 5 (2.5%) | <0.001 |
Superficial Infection, Deep Infection, and Organ Space Infection- these infection types are based on the NSQIP database definition.
Table 5. Change in denominators and numerators from NSQIP to NHSN with effect on SSI rate.
| Hospital | NHSN infections | NHSN colectomy cases | NHSN SSI rate | NSQIP infections | NSQIP colectomy cases | NSQIP SSI rate | Change in SSI rate (NSQIP to NHSN) |
|---|---|---|---|---|---|---|---|
| Hospital 1 | 193 | 1881 | 10.3% | 118 | 1480 | 8.0% | 2.3% |
| Hospital 2 | 141 | 2044 | 6.9% | 210 | 1293 | 16.2% | -9.3% |
| Hospital 3 | 15 | 545 | 2.8% | 34 | 359 | 9.5% | -6.7% |
| Hospital 4 | 38 | 416 | 9.1% | 60 | 365 | 16.4% | -7.3% |
| Hospital 5 | 4 | 163 | 2.5% | 4 | 103 | 3.9% | -1.4% |
| Overall | 391 | 5049 | 7.7% | 426 | 3600 | 11.8% | -4.1% |
Evaluation of the adjusted NHSN and NSQIP reported infection ratings revealed that during the 25 site-time period observations across the collaborative, NSQIP and NHSN ratings matched in only 52% of observations. Twelve of 25 had discordant ratings. For example, Hospital 1 was found to have “exemplary” or “as expected” NSQIP infection ratings (O/E ratios) 100% of the time during each 2012-2014 data collection cycle. During the same time period Hospital 1 was reported to have “Higher” infection ratings 80% of the time from NHSN; see table 6 for further details. There was no correlation between the NSQIP O/E ratio and NHSN SIR with a correlation coefficient of 0.03 (p=0.88), see Figure 3. In addition, there was poor agreement in infection ratings of the two databases across hospitals with a Kappa coefficient of 0.10 (95% CI: -0.14-0.34; p=0.403).
Table 6. NSQIP and NHSN data by cycle.
| 1/1/2012 - 12/31/2012 - NHSN and NSQIP Comparison: Colorectal SSI | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Procedure Volume | Number Infections | Infection Rate | Expected Infection Rate | O.R./S.I.R | 95% C.I. (p value) | Interpretation | ||||||||
| NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | |
| Hospital 1 | 570 | 519 | 57 | 23 | 10.00% | 4.43% | 10.13% | 2.99% | 0.99 | 1.480 | 0.75, 1.29 | 0.961, 2.186 (p=0.073) | As Expected | Same |
| Hospital 2 | 533 | 697 | 96 | 12 | 18.01% | 1.72% | 11.45% | 3.24% | 1.67 | 0.532 | 1.33, 2.10 | 0.288, 0.904 (p=0.017) | Needs Improvement | Lower |
| Hospital 3 | 144 | 250 | 14 | 2 | 9.72% | 0.80% | 8.45% | 2.87% | 1.11 | 0.279 | 0.70, 1.77 | 0.047, 0.921 (p=0.032) | As Expected | Lower |
| Hospital 4 | 140 | 123 | 23 | 5 | 16.43% | 4.07% | 11.05% | 2.95% | 1.43 | 1.380 | 0.95, 2.16 | 0.506, 3.059 (p=0.457) | As Expected | Same |
| Hospital 5 | 30 | 48 | 1 | 0 | 3.33% | 0.00% | 10.54% | 2.77% | 0.77 | 0.000 | 0.39, 1.54 | 2.251, 3.945 (p=0.264) | As Expected | Same |
| 7/1/2012 - 6/30/2013 - NHSN and NSQIP Comparison: Colorectal SSI | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Procedure Volume | Number Infections | Infection Rate | Expected Infection Rate | O.R./S.I.R | 95% C.I. (p value) | Interpretation | ||||||||
| NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | |
| Hospital 1 | 540 | 564 | 46 | 26 | 8.52% | 4.61% | 9.80% | 3.06% | 0.87 | 1.507 | 0.56, 1.00 | 1.006, 2.177 (p=0.047) | As Expected | Higher |
| Hospital 2 | 511 | 669 | 117 | 28 | 22.90% | 4.19% | 11.03% | 3.22% | 2.35 | 1.301 | 1.32, 2.07 | 0.882, 1.856 (p=0.173) | Needs Improvement | Same |
| Hospital 3 | 128 | 166 | 16 | 3 | 12.50% | 1.81% | 8.03% | 2.74% | 1.43 | 0.659 | 0.75, 2.03 | 0.168, 1.794 (p=0.502) | As Expected | Same |
| Hospital 4 | 122 | 123 | 26 | 6 | 21.31% | 4.88% | 10.40% | 3.03% | 1.97 | 1.608 | 1.03, 2.49 | 0.652, 3.345 (p=0.2589) | Needs Improvement | Same |
| Hospital 5 | 30 | 44 | 1 | 1 | 3.33% | 2.27% | 9.84% | 2.83% | 0.78 | 0.803 | 0.41, 1.56 | 0.040, 3.960 (p=0.934) | As Expected | Same |
| 1/1/2013 - 12/31/2013 - NHSN and NSQIP Comparison: Colorectal SSI | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Procedure Volume | Number Infections | Infection Rate | Expected Infection Rate | O.R./S.I.R | 95% C.I. (p value) | Interpretation | ||||||||
| NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | |
| Hospital 1 | 608 | 661 | 44 | 39 | 7.24% | 5.90% | 9.77% | 3.07% | 0.75 | 1.925 | 0.56, 1.00 | 1.388, 2.605 (p=0.00) | Exemplary | Higher |
| Hospital 2 | 547 | 687 | 99 | 38 | 18.10% | 5.53% | 11.53% | 3.27% | 1.66 | 1.690 | 1.32, 2.07 | 1.213, 2.296 (p=0.003) | Needs Improvement | Higher |
| Hospital 3 | 113 | 141 | 12 | 1 | 10.62% | 0.71% | 7.77% | 2.77% | 1.24 | 0.256 | 0.75, 2.03 | 0.013, 1.265 (p=0.12) | As Expected | Same |
| Hospital 4 | 112 | 146 | 20 | 6 | 17.86% | 4.11% | 10.13% | 2.97% | 1.61 | 1.386 | 1.03, 2.49 | 0.562, 2.882 (p=0.4161) | Needs Improvement | Same |
| Hospital 5 | 31 | 44 | 1 | 1 | 3.23% | 2.27% | 9.54% | 2.85% | 0.8 | 0.798 | 0.41, 1.56 | 0.040, 3.938 (p=0.93) | As Expected | Same |
| 7/1/2013 - 6/30/2014 - NHSN and NSQIP Comparison: Colorectal SSI | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Procedure Volume | Number Infections | Infection Rate | Expected Infection Rate | O.R./S.I.R | 95% C.I. (p value) | Interpretation | ||||||||
| NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | |
| Hospital 1 | 608 | 689 | 47 | 39 | 7.73% | 5.66% | 9.29% | 3.04% | 0.75 | 1.864 | 0.56, 1.00 | 1.344, 2.523 (p=0.000) | Exemplary | Higher |
| Hospital 2 | 503 | 654 | 67 | 28 | 13.32% | 4.28% | 11.26% | 3.35% | 1.66 | 1.278 | 1.32, 2.07 | 0.866, 1.823 (p=0.202) | Needs Improvement | Same |
| Hospital 3 | 113 | 130 | 11 | 1 | 9.73% | 0.77% | 7.47% | 2.78% | 1.24 | 0.276 | 0.75, 2.03 | 0.014, 1.362 (P=0.151) | As Expected | Same |
| Hospital 4 | 106 | 134 | 15 | 5 | 14.15% | 3.73% | 9.85% | 2.94% | 1.61 | 1.270 | 1.03, 2.49 | 0.465, 2.814 (p=0.564) | Needs Improvement | Same |
| Hospital 5 | 42 | 54 | 2 | 1 | 4.76% | 1.85% | 9.49% | 2.85% | 0.8 | 0.649 | 0.41, 1.56 | 0.032, 3.200 (p=0.758) | As Expected | Same |
| 1/1/2014 - 12/31/2014 - NHSN and NSQIP Comparison: Colorectal SSI | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Procedure Volume | Number Infections | Infection Rate | Expected Infection Rate | O.R./S.I.R | 95% C.I. (p value) | Interpretation | ||||||||
| NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | NSQIP | NHSN | |
| Hospital 1 | 581 | 633 | 50 | 30 | 8.61% | 4.74% | 8.72% | 3.04% | 0.99 | 1.560 | 0.74, 1.31 | 1.072, 2.199 (p=0.022) | As Expected | Higher |
| Hospital 2 | 480 | 645 | 51 | 24 | 10.63% | 3.72% | 10.95% | 3.40% | 0.97 | 1.096 | 0.73, 1.29 | 0.718, 1.605 (p=0.636) | As Expected | Same |
| Hospital 3 | 115 | 144 | 9 | 4 | 7.83% | 2.78% | 7.24% | 2.82% | 1.05 | 0.985 | 0.53, 1.54 | 0.313, 2.376 (p=1.000) | As Expected | Same |
| Hospital 4 | 121 | 139 | 17 | 6 | 14.05% | 4.32% | 8.92% | 2.85% | 1.46 | 1.516 | 0.92, 2.33 | 0.614, 3.153 (p=0.315) | As Expected | Same |
| Hospital 5 | 47 | 74 | 2 | 3 | 4.26% | 4.05% | 9.73% | 2.75% | 0.75 | 1.472 | 0.38, 1.47 | 0.374, 4.007 (p=0.483) | As Expected | Same |
Figure 3. Scatter plot to depict correlation between NHSN and NSQIP standardized infection findings.
Finally, analysis of the association between SSI rates and case volume for both NHSN and NSQIP found different results. As depicted in Figure 4, the adjusted SSI rate improves in hospitals with higher case volume for the NSQIP database, except for hospital 2. In contrast, the NHSN volume adjusted SIR shown in the same figure reveals a worse infection rate with increasing case volume.
Figure 4. Surgical site infection versus case volume relationship: NSQIP and NHSN findings.
Discussion
Patients, surgeons and payers will all undoubtedly agree that constant efforts to improve the quality of surgical care should be a priority. Incentivizing surgical quality improvement with public recognition programs and financial rewards may be a reasonable motivator for such change, but these incentives should be based on reliable data. In this study, we compared standardized infection rates reported by NHSN and NSQIP across a healthcare system. Despite the hospitals' adherence to the methodology required by both databases, we found no correlation between adjusted SSI rates from the databases. This means that one of the two databases are potentially providing erroneous data. If NHSN data is unreliable, some hospitals are being incorrectly penalized by CMS for the care they are providing.
The NSQIP data is gathered by surgical teams and is discussed by surgical teams in morbidity and mortality conferences. NSQIP has a track record for obtaining highly reliable data.21 Furthermore, the NSQIP database is well adjusted for patient characteristics as well as surgical complexity18. As such, the NSQIP provides information on surgical outcomes that surgeons trust. NHSN data is gathered by infectious disease nurses without surgical expertise. This database has been consistently criticized for its inaccuracies.13, 14 In this study, we also found that within our Partners Colorectal Collaborative the data gathered for this database can reach diametrically opposed conclusions when compared to data collected to measure surgical quality in the NSQIP database. These differences in reported infection rates and their subsequent conclusions appeared to be non-random. The NHSN database information was different due to a number of factors including: 1) problems with infection identification for the numerator, 2) errors in operation identification for the denominator, 3) inadequate adjustment to standardize infection rates across hospitals with the NHSN adjustment not accounting for case complexity, and 4) inadequate benchmarking for comparisons.
The methods used to identify the number of infections following surgery are inherently different between the two databases. There are five methods of data collection to identify SSIs that the CDC has deemed appropriate for the NHSN database.20 This inconsistency makes the NHSN methods for SSI identification unreliable. This was acknowledged by CDC itself in a recent memorandum.22 Despite this, there have not been any changes to the methods for data collection. Institutions who lose patients to follow-up in the postoperative period or use patient surveys, which are likely to not be completed, to collect SSI data will appear to have low infection rates with NHSN. Given the major differences in the mode of SSI identification between the two databases, it is not surprising that NHSN missed several of the NSQIP identified infections. Hospital one was the only hospital where few infections were missed, and it is the only hospital to use the same rigorous method of data collection for both databases.
Quality data collection also requires reliable identification of colectomy cases. The NHSN database uses ICD-9 codes to identify colectomy cases for inclusion. These codes are assigned by hospital billers with no medical training, and have previously been shown to be inaccurate.16, 17 On our review it resulted in the inclusion of cases like cholecystectomy, which was incorrectly coded as colectomy. In addition, the NHSN database is more inclusive than NSQIP and there was an overall increase in cases by 40% in NHSN. These operations are purposefully excluded from the NSQIP database because they inherently have a higher complication rate. Because these types of operations are likely to be complication outliers, it is not possible to reliably adjust for their characteristics. By including these cases, academic centers are likely to be penalized for caring for high risk patients.
Once a numerator and denominator are determined, assessing the quality of a hospital requires risk adjustment. The NSQIP risk-adjustment model to determine SSI rates takes into account multiple patient and operative variables to ensure that the reflected SSI rate accounts for potential differences when comparing hospital quality. The NHSN model only accounts for age, use of general anesthesia, American Society of Anesthesiologists score>2, duration of surgery, open procedure, lack of medical school affiliation, hospital bed size, and wound classification.23, 24 These variables are unlikely to capture all potential confounders of this outcome. NHSN then compares this SIR with data previously reported to the database. There is no accounting for the urgency of cases including in the benchmark year or changes in case-mix at an institution over time. The outcomes reported by NSQIP and NHSN are so different that hospital 1 was reported to be a high quality performers (with low O/E infection rates) according to NSQIP metrics and a low quality performer with consistently “High” infection rates on NHSN metrics. This is likely due to incomplete adjustment for complex operations, patient factors, and inclusion of emergent cases.
This study should be considered in light of several important strengths and limitations. One strength is that this study includes both academic and non-academic hospitals with varying case volume and complexity. This allowed us to see that the NHSN and NSQIP infection rates did not correlate across different types of hospitals. In addition, because we analyzed data at a single hospital network, we were able to obtain more granular information about the specific operations included erroneously in the NHSN database. Limitations of this study are that it is a relatively small sample of hospitals, and all hospitals are located in a single region. A larger and more diverse sample of hospitals may have allowed us to see other trends that might have been missed.
In conclusion, the variation in NHSN infection surveillance methodology leads to inaccurate results and should be better standardized if they are going to be used for hospitals comparisons and for pay-for-performance initiatives. The inadequacy of these data collection methods are readily apparent when NHSN data are compared to data where these processes are standardized. Alternatively, these databases could be integrated rather than being created in parallel to reduce waste and improve the quality of data collected for the CDC; however, this would require creating a mandate that all US hospitals participate in the ACS NSQIP database or something similar. Regardless of what database we ultimately use for the purpose of hospital level comparisons, surgical health services researchers who understand the need to include operative complexity and the effect of surgical indication when adjusting for infection rates should be involved in improving the methods for measuring surgical quality. High quality, reliable data should be used for performance metrics to inform important policy and financial decisions.
Acknowledgments
We would like to acknowledge the work done by Alexandra Koffman, Linda Burr, Jill Steinberg, Marybeth McQueen, and Michael Philips to complete this study.
Disclaimers: None
Funding: This work was supported by the National Cancer Institute at the NIH grant R25CA092203 to Dr. Cauley.
Footnotes
This manuscript will be a podium presentation at the American Society of Colon and Rectal Surgeons conference in Los Angeles, CA: April 30th-May 4th, 2016 (podium #S3).
Contributions: All authors mentioned above made substantial contributions to the conception and design of the project, aided in data acquisition, analysis, and interpretation, and provided critical feedback on the content of the manuscript as well as approval of the final version. Authors LB and CEC co-wrote the article in addition to the previously mentioned contributions.
References
- 1.Mahmoud NN, Turpin RS, Yang G, Saunders WB. Impact of surgical site infections on length of stay and costs in selected colorectal procedures. Surg Infect. 2009;10:539–544. doi: 10.1089/sur.2009.006. [DOI] [PubMed] [Google Scholar]
- 2.Smith RL, Bohl JK, McElearney ST, et al. Wound infection after elective colorectal resection. Ann Surg. 2004;239:599–605. doi: 10.1097/01.sla.0000124292.21605.99. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Aimaq R, Akopian G, Kaufman HS. Surgical site infection rates in laparoscopic versus open colorectal surgery. Am Surg. 2011;77:1290–1294. doi: 10.1177/000313481107701003. [DOI] [PubMed] [Google Scholar]
- 4.Boltz MM, Hollenbeak CS, Julian KG, Ortenzi G, Dillon PW. Hospital costs associated with surgical site infections in general and vascular surgery patients. Surgery. 2011;150:934–942. doi: 10.1016/j.surg.2011.04.006. [DOI] [PubMed] [Google Scholar]
- 5.Tserenpuntsag B, Haley V, Van Antwerpen C, et al. Surgical site infection risk factors identified for patients undergoing colon procedures, New York State 2009-2010. Infect Control Hosp Epidemiol. 2014;35:1006–1012. doi: 10.1086/677156. [DOI] [PubMed] [Google Scholar]
- 6.Bhakta A, Tafen M, Glotzer O, et al. Increased incidence of surgical site infection in IBD patients. Dis Colon Rectum. 2016;59:316–322. doi: 10.1097/DCR.0000000000000550. [DOI] [PubMed] [Google Scholar]
- 7.Bishawi M, Fakhoury M, Denoya PI, Stein S, Bergamaschi R. Surgical site infection rates: open versus hand-assisted colorectal resections. Techniq Coloproct. 2014;18:381–386. doi: 10.1007/s10151-013-1066-z. [DOI] [PubMed] [Google Scholar]
- 8.Drosdeck J, Harzman A, Suzo A, Arnold M, Abdel-Rasoul M, Husain S. Multivariate analysis of risk factors for surgical site infection after laparoscopic colorectal surgery. Surg Endosc. 2013;27:4574–4580. doi: 10.1007/s00464-013-3126-x. [DOI] [PubMed] [Google Scholar]
- 9.Medicaid program: payment adjustment for provider-preventable conditions including health care-acquired conditions. Stat. 2011;76:32816–32838. [PubMed] [Google Scholar]
- 10.Operational Guidance for Reporting Surgical Site Infection (SSI) Data to CDC's NHSN for the Purpose of Fulfilling CMS's Hospital Inpatient Quality Reporting (IQR) Program Requirements. [Last accessed: February 8, 2016];2012 http://www.cdc.gov/nhsn/pdfs/final-ach-ssi-guidance.pdf.
- 11.Garner JS, B J, Scheckler WE, Maki DG, Brachman PS. Proceddings of the International Conference on Nosocomial Infections. Atlanta, GA: Centers for Disease Control and Prevention; 1970. editor Surveillance of nosocomial infections. [Google Scholar]
- 12.Tokars JI, Richards C, Andrus M, et al. The changing face of surveillance for health care-associated infections. Clin Infect Dis. 2004;39:1347–1352. doi: 10.1086/425000. [DOI] [PubMed] [Google Scholar]
- 13.Ju MH, Ko CY, Hall BL, Bosk CL, Bilimoria KY, Wick EC. A comparison of 2 surgical site infection monitoring systems. JAMA Surg. 2015;150:51–57. doi: 10.1001/jamasurg.2014.2891. [DOI] [PubMed] [Google Scholar]
- 14.Haley VB, Van Antwerpen C, Tserenpuntsag B, et al. Use of administrative data in efficient auditing of hospital-acquired surgical site infections, New York State 2009-2010. Infect Control Hosp Epidemiol. 2012;33:565–571. doi: 10.1086/665710. [DOI] [PubMed] [Google Scholar]
- 15.Atchley KD, Pappas JM, Kennedy AT, et al. Use of administrative data for surgical site infection surveillance after congenital cardiac surgery results in inaccurate reporting of surgical site infection rates. Ann Thorac Surg. 2014;97:651–657. doi: 10.1016/j.athoracsur.2013.08.076. [DOI] [PubMed] [Google Scholar]
- 16.Best WR, Khuri SF, Phelan M, et al. Identifying patient preoperative risk factors and postoperative adverse events in administrative databases: results from the Department of Veterans Affairs National Surgical Quality Improvement Program. J Am Coll Surg. 2002;194:257–266. doi: 10.1016/s1072-7515(01)01183-8. [DOI] [PubMed] [Google Scholar]
- 17.Lawson EH, Louie R, Zingmond DS, et al. Using both clinical registry and administrative claims data to measure risk-adjusted surgical outcomes. Ann Surg. 2016;263:50–57. doi: 10.1097/SLA.0000000000001031. [DOI] [PubMed] [Google Scholar]
- 18.Pope G. About ACS NSQIP. [last accessed: 3/30/2016]; [website]. https://www.facs.org/quality-programs/acs-nsqip/about2016.
- 19.Cohen ME, Ko CY, Bilimoria KY, et al. Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. JAm Coll Surg. 2013;217:336–346 e1. doi: 10.1016/j.jamcollsurg.2013.02.027. [DOI] [PubMed] [Google Scholar]
- 20.Procedure-associated Module: Surgical Site Infection (SSI) Event. [Last accessed: February 1, 2016];2015 http://www.cdc.gov/nhsn/pdfs/pscmanual/9pscssicurrent.pdf.
- 21.Hall BL, Hamilton BH, Richards K, Bilimoria KY, Cohen ME, Ko CY. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg. 2009;250:363–376. doi: 10.1097/SLA.0b013e3181b4148f. [DOI] [PubMed] [Google Scholar]
- 22.Beth P, Bell PC. Adherence to the Centers for Disease Control and Prevention's (CDC's) Infection Definitions and Criteria is Needed to Ensure Accuracy, Completeness, and Comparability of Infection Infromation. [Last accessed: February 1, 2016]; http://www.cdc.gov/nhsn/pdfs/cms/nhsn-reporting-signed.pdf.
- 23.Mu Y, Edwards JR, Horan TC, Berrios-Torres SI, Fridkin SK. Improving risk-adjusted measures of surgical site infection for the national healthcare safety network. Infect Control Hosp Epidemiol. 2011;32:970–986. doi: 10.1086/662016. [DOI] [PubMed] [Google Scholar]
- 24.Berrios-Torres SI, Mu Y, Edwards JR, Horan TC, Fridkin SK. Improved risk adjustment in public reporting: coronary artery bypass graft surgical site infections. Infect Control Hosp Epidemiol. 2012;33:463–469. doi: 10.1086/665313. [DOI] [PubMed] [Google Scholar]