Supplemental Digital Content is available in the text.
Background:
Both the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) and the American Society of Plastic Surgeons Tracking Operations and Outcomes for Plastic Surgeons (TOPS) databases track 30-day outcomes.
Methods:
Using the 2008–2016 TOPS and NSQIP databases, we compared patient characteristics and postoperative outcomes for 5 common plastic surgery procedures. A weighted TOPS population was used to mirror the NSQIP population in clinical and demographic characteristics to compare postoperative outcomes.
Results:
We identified 154,181 cases. Compared with NSQIP patients, TOPS patients were more likely to be younger (47.9 versus 50.0 years), have American Society of Anesthesiologists class I-II (92.1% versus 74.6%), be outpatient (66.0% versus 49.3%), and be smokers (18.7% versus 11.7%). TOPS had extensive missing data: body mass index (40.6%), American Society of Anesthesiologists class (34.9%), diabetes (39.3%), and smoking status (37.2%). NSQIP was missing <1% of all shared categories except race (15.6%). The entire TOPS cohort versus only TOPS patients without missing data had higher rates of dehiscence (5.1% versus 3.5%) and infection (2.1% versus 1.7%). TOPS versus NSQIP patients had higher dehiscence rates (5.1% versus 1.0%) but lower rates of return to the operating room (3.1% versus 6.6%), infection (2.1% versus 3.0%), and medical complications (0.3% versus 2.2%). Nonweighted and weighted TOPS cohorts had similar 30-day outcomes.
Conclusions:
NSQIP and TOPS populations are different in characteristics and outcomes, likely due to differences in collection methodology and the types physicians using the databases. The strengths of each dataset can be used together for research and quality improvement.
INTRODUCTION
National databases collect and track patient information such as demographic data, administrative and cost data, medical history, health status, test results, procedures performed, and health outcomes in a given healthcare system.1–3 Analysis of these data is used to direct policy and improve quality of care and patient outcomes.4–7 The content and quality of databases vary, making it critical for physicians and researchers to understand the populations and specific data that a database contains. When used appropriately, databases have the power to influence positive change across healthcare systems.1,4,8–11
National databases are commonly used for surgical outcomes research. A large body of work in plastic surgery has used the American College of Surgeons—National Surgical Quality Improvement Program (NSQIP) database.12–18 An alternative database, the American Society of Plastic Surgeons (ASPS)—Tracking Operations and Outcomes for Plastic Surgeons (TOPS) has recently been considered for outcomes research specifically in plastic surgery but has a less well-understood patient population.19,20
There are several key differences between the NSQIP and TOPS. First, cases are sampled randomly for the NSQIP, whereas ASPS members have the ability to choose the cases they contribute to TOPS. Entry of certain demographic information into TOPS is not required. Lastly, NSQIP data are abstracted by a third party, whereas in TOPS, data are entered by the surgeons themselves, which can then lead to bias.
To better understand the differences between NSQIP and TOPS, we compared data from each of the databases for a variety of plastic surgery procedures. The goal of this study was to compare common plastic surgery procedures available in both datasets for differences in patient characteristics, missing data, and outcomes with the intent to inform future researchers on the characteristics of the populations within each dataset, the differences in complication rate reporting, and how to use each database to produce reliable and valid evidence.
METHODS
Data and Study Cohort
Our comparison of TOPS and NSQIP involved patients 18–89 years of age undergoing 1 of 5 procedures between 2008 and 2016. These procedures were identified using Current Procedural Terminology (CPT) codes and include breast reduction (19318), prosthetic breast reconstruction (19357, 19342), autologous breast reconstruction (19361, 19364, 19367), free-flap procedures (15756, 15757, 15758, 20969, 20955), and pressure ulcer repair (15922, 15934, 15936, 15944, 15946, 15952, 15956). From each dataset, we collected a case for each surgery type if one of the above-mentioned CPT codes was recorded in any CPT field.
The NSQIP is a validated and nationally representative database designed for 30-day surgical outcomes research across a broad spectrum of surgical specialties. In brief, NSQIP data are currently collected at over 600 hospitals in 49 states across the United States.21 At participating sites, surgically trained nurses collect data from patient medical charts on randomly assigned cases from all surgical specialties. The data recorded include case type, hospital characteristics, patient preoperative demographic and clinical characteristics, intraoperative characteristics, and 30-day surgical outcomes. Importantly, the NSQIP can be viewed as a nationally representative random sample of surgical cases. However, the NSQIP lacks complication reporting for many specific outcomes of interest for plastic surgeons, such as loss of graft, flap, or prosthesis.12 The TOPS database is designed to track 30-day surgical outcomes for plastic surgery patients, specifically. Implemented initially in 2002, TOPS is contributed to by over 700 ASPS member surgeons across the United States. Surgeons enter data directly for the TOPS dataset using an electronic data capture interface, where they are prompted to enter case type and 30-day surgical outcomes, as well as patient demographic and clinical information.22
The 2 primary goals of our analysis were to compare patient characteristics and surgical complication rates across the 2 datasets. The 2 datasets both collect the following patient characteristics: age, body mass index (BMI), smoking status, diabetes status, inpatient/outpatient surgery status, and American Society of Anesthesiologists (ASA) classification score. Regarding the surgical outcomes reported by both datasets, we compared medical complications (cardiac arrest, myocardial infarction, stroke/cerebrovascular accident (CVA) occurrence, renal failure, organ space infection, systemic shock, systemic sepsis, pneumonia, reintubation, or urinary infection), wound dehiscence, mortality, venous thromboembolism (deep venous thrombosis [DVT] or pulmonary embolism), return to operating room (OR), and surgical infection (progressive renal insufficiency, superficial surgical site infection, and wound infection). We restricted our cohort to those with documented BMI >10 and <100. Our study protocol was reviewed by the University of Utah Institutional Review Board and was given an exempt status.
Statistical Analyses
Patient characteristics and 30-day outcomes were compared across the 2 datasets. In each, descriptive statistics are provided, where we give mean and SD for continuous variables and counts and percentages for categorical variables. Hypothesis-driven tests are provided for most comparisons. χ2 tests or exact tests were used for categorical variable comparisons, where appropriate. T tests were used to compare continuous covariates. In analyses with weighted populations, we used survey-weighted χ2 tests or survey-weighted exact tests.
In our first analysis, we compare patient characteristics across datasets within each procedure category for all available information within a given variable. As has been discussed in previous research, TOPS suffers from extensive missing data.20 With the goal of generalizing TOPS surgical complications to the population represented by NSQIP, we sought to eliminate patients with any field of missing demographic or preoperative clinical information from the analysis. To understand bias induced by including only complete TOPS cases, we summarized the extent of missing data across the datasets. We then compared complications in the entire TOPS cohort versus the TOPS cohort with complete data and compared complications in the complete TOPS dataset to the complete NSQIP patients. Within the complete TOPS cases, we sought to compare complication rates in a weighted TOPS sample. The weights were computed to generate a pseudo-population of TOPS patients, where TOPS and NSQIP are as close as possible to being identical in distribution with respect to all patient characteristics discussed above. We chose to use inverse probability of selection weights (IPSWs), which are proposed in the literature, where the intent is to generalize results from randomized controlled trials to a target population.23 We developed a model for the IPSW within each surgical category because we felt each surgical category represented potentially unique patient populations. In each case, the selection model involved a multivariable logistic regression on all preoperative patient variables shared across the datasets (each variable in Table 1), where interaction and higher order terms were included if the resultant model improved the balance in covariates across datasets. These variables were included because we felt each variable has the potential to influence complications and is shown to differ across the datasets in distribution. Balance was assessed in each case using a weighted absolute standardized mean difference. Once IPSWs were generated for the TOPS dataset, complications in the weighted TOPS dataset were compared with those in the unweighted NSQIP. All analyses were performed using RStudio Version 1.1.383 (2009–2017 RStudio, Inc., Boston, Mass.).
Table 1.
Crude Data Comparisons of Preoperative Patient Characteristics
| Overall | Breast Reductions | Prosthetic Reconstruction | Autologous Reconstruction | Free Flap | Pressure Ulcer Repair | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NSQIP | TOPS | P | NSQIP | TOPS | P | NSQIP | TOPS | P | NSQIP | TOPS | P | NSQIP | TOPS | P | NSQIP | TOPS | P | |
| N | 94,823 | 59,358 | 26,012 | 29,746 | 48,219 | 22,749 | 14,289 | 4787 | 5085 | 1017 | 1218 | 1059 | ||||||
| Age, mean (SD) | 50.00 (12.69) | 47.85 (13.31) | <0.001 | 44.96 (14.49) | 43.80 (14.07) | <0.001 | 51.12 (10.96) | 51.87 (10.85) | <0.001 | 51.90 (9.89) | 51.59 (9.80) | 0.062 | 58.79 (14.72) | 52.44 (16.85) | <0.001 | 54.77 (16.31) | 51.83 (16.18) | <0.001 |
| ASA class, n (%) | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | ||||||||||||
| 1 | 11,198 (11.8) | 14,613 (37.8) | 6275 (24.1) | 10,439 (50.3) | 4097 (8.5) | 3508 (24.4) | 706 (4.9) | 483 (20.1) | 117 (2.3) | 92 (19.9) | 3 (0.2) | 91 (14.1) | ||||||
| 2 | 59,467 (62.8) | 20,957 (54.3) | 15,688 (60.4) | 9304 (44.8) | 33,233 (69.0) | 9537 (66.4) | 9008 (63.1) | 1682 (70.0) | 1412 (27.8) | 217 (47.0) | 126 (10.4) | 217 (33.5) | ||||||
| 3 | 23,194 (24.5) | 2954 (7.6) | 3950 (15.2) | 999 (4.8) | 10,718 (22.2) | 1283 (8.9) | 4517 (31.6) | 236 (9.8) | 3204 (63.1) | 142 (30.7) | 805 (66.3) | 294 (45.4) | ||||||
| 4 | 863 (0.9) | 86 (0.2) | 72 (0.3) | 13 (0.1) | 123 (0.3) | 17 (0.1) | 46 (0.3) | 3 (0.1) | 342 (6.7) | 11 (2.4) | 280 (23.1) | 42 (6.5) | ||||||
| 5 | 2 (0.0) | 15 (0.0) | 1 (0.0) | 1 (0.0) | 0 (0.0) | 11 (0.1) | 0 (0.0) | 0 (0.0) | 1 (0.0) | 0 (0.0) | 0 (0.0) | 3 (0.5) | ||||||
| BMI, mean (SD) | 28.68 (6.57) | 28.72 (6.12) | 0.334 | 31.25 (6.44) | 30.11 (5.99) | <0.001 | 27.40 (6.36) | 26.85 (5.85) | <0.001 | 29.12 (5.97) | 28.47 (5.53) | <0.001 | 26.91 (6.46) | 27.67 (6.35) | 0.041 | 27.29 (7.68) | 27.58 (7.64) | 0.558 |
| Diabetes (preop), n (%) | 5989 (6.3) | 1459 (4.0) | <0.001 | 1483 (5.7) | 639 (3.3) | <0.001 | 2666 (5.5) | 581 (4.2) | <0.001 | 937 (6.6) | 118 (5.1) | 0.008 | 613 (12.1) | 28 (7.7) | 0.015 | 290 (23.8) | 93 (18.3) | 0.015 |
| Outpatient, n (%) | 46,721 (49.3) | 39,139 (66.0) | <0.001 | 22,595 (86.9) | 24,841 (83.5) | <0.001 | 22,945 (47.6) | 13,451 (59.3) | <0.001 | 910 (6.4) | 614 (12.9) | <0.001 | 144 (2.8) | 58 (5.7) | <0.001 | 127 (10.4) | 175 (16.5) | <0.001 |
| Male sex, n (%) | 4514 (4.8) | 2001 (3.4) | <0.001 | 330 (1.3) | 468 (1.6) | 0.004 | 95 (0.2) | 126 (0.6) | <0.001 | 49 (0.3) | 29 (0.6) | 0.02 | 3237 (63.7) | 647 (63.6) | 0.997 | 803 (65.9) | 731 (69.0) | 0.126 |
| Smoker status, n (%) | 11,082 (11.7) | 6976 (18.7) | <0.001 | 2489 (9.6) | 3014 (15.2) | <0.001 | 5550 (11.5) | 3081 (21.9) | <0.001 | 1372 (9.6) | 494 (20.8) | <0.001 | 1407 (27.7) | 194 (47.5) | <0.001 | 264 (21.7) | 193 (33.6) | <0.001 |
| Race, n (%) | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | ||||||||||||
| White | 64,563 (81.9) | 43,089 (83.0) | 13,044 (71.5) | 20,918 (79.3) | 36,931 (86.3) | 18,054 (88.0) | 10,022 (80.7) | 2910 (81.8) | 3766 (87.5) | 605 (84.7) | 800 (73.6) | 602 (78.1) | ||||||
| Black | 11,124 (14.1) | 5464 (10.5) | 4853 (26.6) | 3782 (14.3) | 3819 (8.9) | 1183 (5.8) | 1834 (14.8) | 317 (8.9) | 374 (8.7) | 59 (8.3) | 244 (22.4) | 123 (16.0) | ||||||
| Hispanic | 1177 (1.5) | 2567 (4.9) | 356 (1.9) | 1467 (5.6) | 600 (1.4) | 817 (4.0) | 181 (1.4) | 223 (6.3) | 30 (0.7) | 32 (4.5) | 10 (0.9) | 28 (3.6) | ||||||
| Asian | 2594 (3.3) | 652 (1.3) | 231 (1.3) | 152 (0.6) | 1761 (4.1) | 396 (1.9) | 467 (3.8) | 80 (2.2) | 121 (2.8) | 17 (2.4) | 14 (1.3) | 7 (0.9) | ||||||
| Native American or Pacific Islander | 551 (0.7) | 151 (0.3) | 109 (0.6) | 43 (0.2) | 279 (0.7) | 67 (0.3) | 90 (0.7) | 29 (0.8) | 44 (1.0) | 1 (0.1) | 29 (2.7) | 11 (1.4) | ||||||
Results
There were 94,823 cases identified in NSQIP and 59,358 in TOPS meeting our inclusion criteria. Across all 5 procedure categories, TOPS patients were younger (47.9 years versus 50.0 years), more likely to have an ASA class of I or II (92.1% versus 74.6%), more likely to be outpatient (66.0% versus 49.3%), and more likely to be smokers (18.7% versus 11.7%). NSQIP patients were more likely to have diabetes (6.3% versus 4.0%) and more likely to be men (4.8% versus 3.4%). These trends were similar across the 4 procedure groups with few exceptions, such as a slightly older age in TOPS patients undergoing prosthetic breast reconstruction (51.87 years versus 51.12 years) (Table 1).
Table 2 demonstrates the extent of missing data across the 2 datasets and 5 procedure groups. In most categories, the TOPS database had far greater amounts of missing data overall, specifically with regard to age (6.3% versus 0.1%), BMI (40.6% versus 0.8%), ASA class (34.9% versus 0.1%), diabetes status (39.3% versus 0.0%), and smoking status (37.2% versus 0.0%). Within TOPS, free flaps had the greatest percentage of missing data in each of those categories. NSQIP had a higher rate of missing data overall in one category: race (15.6% versus 12.5%). Missing data on race, however, were not higher in NSQIP across all procedures; they were only higher in the breast reduction (28.5 % versus 11.4%) and prosthetic breast reconstruction (10.0% versus 9.8%) cohorts (Table 2).
Table 2.
Summary of Missing Data across Datasets by Surgery Type
| Overall | Breast Reductions | Prosthetic Reconstruction | Autologous Reconstruction | Free Flap | Pressure Ulcer Repair | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NSQIP | TOPS | NSQIP | TOPS | NSQIP | TOPS | NSQIP | TOPS | NSQIP | TOPS | NSQIP | TOPS | |
| Missing age | 48 (0.1) | 3761 (6.3) | 5 (0.0) | 2130 (7.2) | 4 (0.0) | 1169 (5.1) | 0 (0.0) | 257 (5.4) | 23 (0.5) | 121 (11.9) | 16 (1.3) | 84 (7.9) |
| Missing ASA | 99 (0.1) | 20,733 (34.9) | 26 (0.1) | 8990 (30.2) | 48 (0.1) | 8393 (36.9) | 12 (0.1) | 2383 (49.8) | 9 (0.2) | 555 (54.6) | 4 (0.3) | 412 (38.9) |
| Missing BMI | 717 (0.8) | 24,084 (40.6) | 238 (0.9) | 10,925 (36.7) | 311 (0.6) | 9584 (42.1) | 60 (0.4) | 2141 (44.7) | 39 (0.8) | 698 (68.6) | 69 (5.7) | 736 (69.5) |
| Missing diabetes status | 0 (0.0) | 23,318 (39.3) | 0 (0.0) | 10,615 (35.7) | 0 (0.0) | 9038 (39.7) | 0 (0.0) | 2462 (51.4) | 0 (0.0) | 652 (64.1) | 0 (0.0) | 551 (52.0) |
| Missing admission type | 0 (0.0) | 84 (0.1) | 0 (0.0) | 7 (0.0) | 0 (0.0) | 60 (0.3) | 0 (0.0) | 17 (0.4) | 0 (0.0) | 0 | 0 | 0 |
| Missing race | 14,814 (15.6) | 7435 (12.5) | 7419 (28.5) | 3384 (11.4) | 4829 (10.0) | 2232 (9.8) | 1695 (11.9) | 1228 (25.7) | 750 (14.7) | 303 (29.8) | 121 (9.9) | 288 (27.2) |
| Missing smoker status | 1 (0.0) | 22,068 (37.2) | 0 (0.0) | 9910 (33.3) | 1 (0.0) | 8654 (38.0) | 0 (0.0) | 2410 (50.3) | 0 (0.0) | 609 (59.9) | 0 (0.0) | 485 (45.8) |
| Missing sex | 75 (0.1) | 0 (0.0) | 12 (0.0) | 0 (0.0) | 50 (0.1) | 0 (0.0) | 11 (0.1) | 0 (0.0) | 2 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
Patients in TOPS with complete data were compared with those with missing data with respect to all surgical complication rates of interest. Overall, patients with complete data had higher rates of dehiscence (5.1% versus 3.5%), surgical infection (2.1% versus 1.7%), medical complication (0.3% versus 0.2%), and partial flap loss (1.2% versus 0.8%) than those with missing data. Where applicable, these trends were similar across all procedures (Table 3).
Table 3.
Comparing Complications in TOPS-EC versus TOPS-WM
| Overall | Breast Reductions | Prosthetic Reconstruction | Autologous Reconstruction | Free Flap | Pressure Ulcer Repair | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TOPS-EC | TOPS-WM | TOPS-EC | TOPS-WM | TOPS-EC | TOPS-WM | TOPS-EC | TOPS-WM | TOPS-EC | TOPS-WM | TOPS-EC | TOPS-WM | |
| N | 59,358 | 29,177 | 29,746 | 15,591 | 22,749 | 11,315 | 4787 | 1853 | 1017 | 208 | 1059 | 210 |
| Medical complications | 110 (0.2) | 79 (0.3) | 24 (0.1) | 17 (0.1) | 69 (0.3) | 47 (0.4) | 9 (0.2) | 9 (0.5) | 3 (0.3) | 2 (1.0) | 5 (0.5) | 4 (1.9) |
| Dehiscence | 2080 (3.5) | 1481 (5.1) | 1105 (3.7) | 863 (5.5) | 551 (2.4) | 372 (3.3) | 285 (6.0) | 194 (10.5) | 31 (3.0) | 12 (5.8) | 108 (10.2) | 40 (19.0) |
| Mortality | 18 (0.0) | 1 (0.0) | 3 (0.0) | 1 (0.0) | 2 (0.0) | 0 (0.0) | 4 (0.1) | 0 (0.0) | 3 (0.3) | 0 (0.0) | 6 (0.6) | 0 (0.0) |
| Venous thromboembolism | 71 (0.1) | 34 (0.1) | 37 (0.1) | 17 (0.1) | 19 (0.1) | 12 (0.1) | 11 (0.2) | 5 (0.3) | 3 (0.3) | 0 (0.0) | 1 (0.1) | 0 (0.0) |
| Return to OR | 1855 (3.1) | 908 (3.1) | 419 (1.4) | 222 (1.4) | 1011 (4.4) | 512 (4.5) | 311 (6.5) | 145 (7.8) | 85 (8.4) | 23 (11.1) | 29 (2.7) | 6 (2.9) |
| Surgical infection | 1003 (1.7) | 601 (2.1) | 344 (1.2) | 243 (1.6) | 495 (2.2) | 281 (2.5) | 131 (2.7) | 64 (3.5) | 23 (2.3) | 9 (4.3) | 10 (0.9) | 4 (1.9) |
| Total flap loss | 155 (0.3) | 53 (0.2) | NA | NA | NA | NA | 84 (1.8) | 33 (1.8) | 46 (4.5) | 14 (6.7) | 2 (0.2) | 1 (0.5) |
| Partial flap loss | 490 (0.8) | 351 (1.2) | NA | NA | NA | NA | 126 (2.6) | 87 (4.7) | 19 (1.9) | 9 (4.3) | 13 (1.2) | 7 (3.3) |
EC, entire cohort; NA, not available; WM, without missing.
When comparing patients with complete data across the 2 databases (unweighted), NSQIP was found to have higher rates of medical complications (2.2% versus 0.3%), mortality (0.1% versus 0.0%), venous thromboembolism (0.5 versus 0.1%), return to OR (6.6% versus 3.1%), and surgical infection (3.0% versus 2.1%). TOPS was found to have higher rates of dehiscence (5.1% versus 1.0%). These trends were similar across the procedures, with the largest percentage differences between NSQIP and TOPS seen in pressure ulcer repair (15.3% versus 1.9% medical complications and 3.5% versus 19.0% dehiscence) (Table 4).
Table 4.
Comparisons of Complications across Datasets for Patients with Complete Demographic/Clinical Data
| Overall | Breast Reductions | Prosthetic Reconstruction | Autologous Reconstruction | Free Flap | Pressure Ulcer Repair | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NSQIP-WM | TOPS-WM | P | NSQIP-WM | TOPS-WM | P | NSQIP-WM | TOPS-WM | P | NSQIP-WM | TOPS-WM | P | NSQIP-WM | TOPS-WM | P | NSQIP-WM | TOPS-WM | P | |
| N | 79,399 | 29,177 | 18,463 | 15,591 | 43,081 | 11,315 | 12,541 | 1853 | 4285 | 208 | 1029 | 210 | ||||||
| Medical complications | 1758 (2.2) | 79 (0.3) | <0.001 | 118 (0.6) | 17 (0.1) | <0.001 | 603 (1.4) | 47 (0.4) | <0.001 | 325 (2.6) | 9 (0.5) | <0.001 | 555 (13.0) | 2 (1.0) | <0.001 | 157 (15.3) | 4 (1.9) | <0.001 |
| Dehiscence | 798 (1.0) | 1481 (5.1) | <0.001 | 126 (0.7) | 863 (5.5) | <0.001 | 261 (0.6) | 372 (3.3) | <0.001 | 194 (1.5) | 194 (10.5) | <0.001 | 181 (4.2) | 12 (5.8) | 0.369 | 36 (3.5) | 40 (19.0) | <0.001 |
| Mortality | 85 (0.1) | 1 (0.0) | <0.001 | 3 (0.0) | 1 (0.0) | 0.739 | 5 (0.0) | 0 (0.0) | NA | 4 (0.0) | 0 (0.0) | NA | 40 (0.9) | 0 (0.0) | NA | 33 (3.2) | 0 (0.0) | NA |
| Venous thromboembolism | 411 (0.5) | 34 (0.1) | <0.001 | 35 (0.2) | 17 (0.1) | 0.079 | 164 (0.4) | 12 (0.1) | <0.001 | 127 (1.0) | 5 (0.3) | 0.003 | 77 (1.8) | 0 (0.0) | NA | 8 (0.8) | 0 (0.0) | NA |
| Return to OR | 5225 (6.6) | 908 (3.1) | <0.001 | 397 (2.2) | 222 (1.4) | <0.001 | 2623 (6.1) | 512 (4.5) | <0.001 | 1332 (10.6) | 145 (7.8) | <0.001 | 772 (18.0) | 23 (11.1) | 0.013 | 101 (9.8) | 6 (2.9) | 0.002 |
| Surgical infection | 2406 (3.0) | 601 (2.1) | <0.001 | 419 (2.3) | 243 (1.6) | <0.001 | 976 (2.3) | 281 (2.5) | 0.181 | 535 (4.3) | 64 (3.5) | 0.116 | 413 (9.6) | 9 (4.3) | 0.015 | 63 (6.1) | 4 (1.9) | 0.022 |
NA, not available; WM, without missing.
When inverse probability of selection weighting was applied to each surgical cohort considered in the TOPS dataset, there was continued discordance in complication rates when comparing the weighted TOPS samples to the NSQIP samples. As will be discussed further, weighting was used to mitigate differences between the datasets with respect to all patient characteristics shared across the datasets. Overall and within each surgery type, rates of medical complication were higher in the NSQIP than in the weighted TOPS, and the differences were statistically significant. In the overall comparison, mortality, venous thromboembolism, return to the OR, and surgical infection rates remained higher in the NSQIP. In certain surgical categories, the rates were higher in the NSQIP but not associated with statistically significant P values. Rates of dehiscence remained higher in the weighted TOPS sample when compared with the NSQIP. Only in the free-flap cohort was the difference in dehiscence not associated with a P value significant at any reasonable level.
DISCUSSION
The use of clinical registries is vital to the quality improvement efforts of our healthcare system by contributing to current and agreed upon care guidelines,8 evaluating disease-specific outcomes4,24 and reducing healthcare costs.25–27 Blind use of data from any given registry is unadvisable. A keen understanding of the patient populations within each registry is vital to making inferences about surgical outcomes.12,19,28–30 The TOPS and NSQIP databases are 2 robust national databases that can be used for plastic surgery outcomes. Our results demonstrate that the TOPS and NSQIP databases have differing patient populations, which must be understood when using the data for analysis.
As has been highlighted in previous literature, the NSQIP and TOPS exhibit unique strengths and weaknesses.20,28,29 The validity of the NSQIP has been documented extensively. It has been shown that there is very limited (<2%) disagreement among data collectors and auditors of the NSQIP when considering nearly all of the multiple hundred variables included in the dataset.3,20,31,32 Moreover, rates of surgical outcomes have been validated in single-institution studies.31 Importantly, the NSQIP sampling methodology produces what can be considered a random sample of surgical cases conducted at US hospitals, making results interpretable and generalizable to a national population of surgical cases.21 Though not used nearly as extensively, recent research has validated outcomes in TOPS through cross-validation with CosmetAssure and the NSQIP database across several procedures.20,33
Although surgically trained nurses collect surgical cases for the NSQIP, board-certified members of ASPS contribute their own data to TOPS through an electronic data capture. TOPS is beneficial to plastic surgery–specific research because it is not restricted to data collected at hospitals and because many more complications specific to plastic surgery are recorded, such as flap, graft, or prosthesis loss. As such, with many plastic surgery procedures being performed in smaller clinics and outside major hospitals (eg, breast reduction and prosthetic breast reconstruction), it is likely that TOPS more accurately represents the patient populations undergoing such procedures and tracks relevant outcomes.20,22,34 Furthermore, plastic surgeons may be able to more accurately identify and document postoperative wound-related complications when compared with nurses abstracting data from the medical record. However, because of the nature of data collection in TOPS, there is no guarantee against selective reporting of cases or outcomes. Along with differences in the patient populations between datasets, it is possible that complications are underreported in TOPS. TOPS represents a potentially impactful source for research in plastic surgery, but its cohort and contents need to be well understood to draw generalizable conclusions.
We found considerable differences in the distribution of patient characteristics across NSQIP and TOPS overall and within each surgical category. Much of the discordance in patient characteristics is understandable given our knowledge of the sampling methods in each database. In the NSQIP, patients had higher ASA classification, were older, had a higher BMI, were more often diabetic, and were less often having procedures performed on an outpatient basis. Because cases are recorded at hospitals alone in the NSQIP, it is reasonable that the NSQIP patients appear to have lower health status with respect to available measures. TOPS patients may appear healthier because many of their cases are being performed in clinics and on an outpatient basis. Interestingly, overall and across all surgery types, TOPS patients appear to more often be smokers and white. Broadly speaking, it is evident that the NSQIP and TOPS represent distinct patient populations. Researchers must be cautious about interpreting results and must carefully define the target populations with which they intend to study when using TOPS, in particular.
A noteworthy drawback of the TOPS database is the extent to which patient characteristics are missing or not recorded. For each surgery type, at least one patient characteristic was missing by >33%. The missingness across patient characteristics was as severe as 69.5% of pressure ulcer patients missing BMI information. Meanwhile, only race was missing extensively in the NSQIP, where the issue was most severe among breast reduction patients. For research limited entirely to descriptive purposes, missing data may not be of concern. However, for any analysis intending to ascribe associations or causation, adjustment for demographic and clinical information in statistical methods is paramount to limiting effect estimate bias. If attempting to use TOPS for such research, statisticians should be aware of the extensive missing information. The issue of missing data would be of less concern if key surgical outcomes of interest were of equal prevalence between those with missing data and those with complete clinical and demographic information, and if we knew the patients with complete data did not differ from those of the general population captured by TOPS in terms of health and demographics. However, as is displayed in Table 3, many complication rates differ between the complete and missing data TOPS samples. Since the extent of missing data is too large for available missing data methods (eg, imputation-based methods), researchers must proceed with additional caution about describing cohorts obtained using TOPS subjects with complete information.
The TOPS and the NSQIP databases clearly exhibit distinct patient populations and confront differing degrees of missing data. We sought to better understand whether complication reporting was similar across the 2 datasets and if any differences in reporting could be explained away by mitigating differences between the samples with respect to patient characteristics. For those with complete data, if complication rates could be made comparable in a weighted analysis using the NSQIP as a target population, then it is arguable that such a weighted TOPS could be used for nationally generalizable analyses.
Among complete cases in the raw data, we saw that a larger proportion of patients experienced all complications except mortality and dehiscence in the NSQIP. Mortality was comparable across the 2 datasets, but dehiscence rates were higher in TOPS. Although smokers are more prevalent in TOPS, NSQIP subjects were less healthy with respect to ASA classification score, BMI, and diabetes status. Given that these characteristics are known risk factors for various complications, we would expect complication rates to be higher in the NSQIP.35,36 However, differences in patient health were mitigated in our weighted analysis. We used inverse probability of selection weights to balance the TOPS population toward the NSQIP population in all variables measured. (See tables, Supplemental Digital Content 1, which displays the balance in covariates achieved between the NSQIP and weighted TOPS for all surgeries, http://links.lww.com/PRSGO/B384.) Only for race in the free flap patients was the distribution not satisfactorily similar between the weighted TOPS and NSQIP databases. Otherwise we achieved an appreciable level of balance in covariates after weighting for all surgical categories. Table 5 indicates that even when the distribution of all measured patient characteristics in TOPS are weighted to emulate that of the NSQIP, complication rates grew more divergent. As such, it is not clear that the differences in patient populations explain complication rates.
Table 5.
Comparisons of Complications across Datasets for Complete NSQIP and Propensity-weighted Complete TOPS
| Overall | Breast Reductions | Prosthetic Reconstruction | Autologous Reconstruction | Free Flap | Pressure Ulcer Repair | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NSQIP-WM | Weighted TOPS | P | NSQIP-WM | Weighted TOPS | P | NSQIP-WM | Weighted TOPS | P | NSQIP-WM | Weighted TOPS | P | NSQIP-WM | Weighted TOPS | P | NSQIP-WM | Weighted TOPS | P | |
| N | 79,399 | 29,053.7 | 18,463 | 15,647.6 | 43,081 | 11,396.2 | 12,541 | 1833.7 | 4285 | 191.9 | 1029 | 211 | ||||||
| Medical complications | 1758 (2.2) | 121.4 (0.42) | <0.001 | 118 (0.6) | 22.5 (0.14) | <0.001 | 603 (1.4) | 54.0 (0.47) | <0.001 | 325 (2.6) | 8.1 (0.44) | <0.001 | 555 (13.0) | 1.5 (0.79) | <0.001 | 157 (15.3) | 7.2 (3.43) | 0.003 |
| Dehiscence | 798 (1.0) | 1701.9 (5.86) | <0.001 | 126 (0.7) | 946.3 (6.05) | <0.001 | 261 (0.6) | 416.9 (3.66) | <0.001 | 194 (1.5) | 210.9 (11.50) | <0.001 | 181 (4.2) | 8.5 (4.44) | 0.889 | 36 (3.5) | 35.5 (16.84) | <0.001 |
| Mortality | 85 (0.1) | 1.2 (0.00) | <0.001 | 3 (0.0) | 1.1 (0.01) | 0.469 | 5 (0.0) | 0.0 (0.00) | NA | 4 (0.0) | 0.0 (0.00) | NA | 40 (0.9) | 0.0 (0.00) | NA | 33 (3.2) | 0.0 (0.00) | NA |
| Venous thromboembolism | 411 (0.5) | 37.4 (0.13) | <0.001 | 35 (0.2) | 17.3 (0.11) | 0.080 | 164 (0.4) | 15.8 (0.14) | 0.001 | 127 (1.0) | 3.9 (0.21) | <0.001 | 77 (1.8) | 0.0 (0.00) | NA | 8 (0.8) | 0.0 (0.00) | NA |
| Return to OR | 5225 (6.6) | 1121.1 (3.86) | <0.001 | 397 (2.2) | 233.8 (1.49) | <0.001 | 2623 (6.1) | 542.8 (4.76) | <0.001 | 1332 (10.6) | 154.6 (8.43) | 0.021 | 772 (18.0) | 24.8 (12.93) | 0.198 | 101 (9.8) | 9.8 (4.64) | 0.134 |
| Surgical infection | 2406 (3.0) | 704.8 (2.43) | <0.001 | 419 (2.3) | 271.0 (1.73) | 0.002 | 976 (2.3) | 318.3 (2.79) | 0.007 | 535 (4.3) | 63.9 (3.48) | 0.167 | 413 (9.6) | 11.5 (5.97) | 0.271 | 63 (6.1) | 4.9 (2.32) | 0.174 |
Datasets have different strengths, particularly in reporting certain complications (Table 6). The TOPS database could be improved with better surgeon involvement and more complete data input. This could be achieved by incentivizing participation or mandating fields to be completed before case submission. Such efforts should be carefully taken to ensure that participation rates do not decline. Our data can also be used by society leadership to advocate for improvement in the NSQIP or other existing databases to include more plastic surgery–specific outcomes.
Table 6.
Advantages of TOPS and NSQIP Databases
| TOPS | NSQIP |
|---|---|
| Tailored for plastic surgery | Data collected by trained surgical nurses |
| Captures academic, community, and private practice settings | Auditing to evaluate data reliability |
| Low financial requirement | Low disagreement rate between reviewers |
| Better captures plastic surgery–specific variables and outcomes | Captures more demographic data |
| Allows for additional modules and procedure-specific variables | Has expanded to include 8 additional specialties |
| Captures BREAST-Q scores | Completeness of data |
| Vast number of surgical procedures |
CONCLUSIONS
The populations represented by the TOPS and NSQIP databases are significantly different. Such differences may stem from the fact that NSQIP data come from large hospitals, and the TOPS data are largely from private physicians and small groups. TOPS is missing a large amount of patient characteristics data. When assessing only TOPS patients with complete data, most rates of complications increase. Differences in complication rates may also be explained by differences in data collection methodology. After mitigating differences in patient characteristics via a weighted analysis of TOPS, complication rates remained discordant between the 2 sources. Only for dehiscence were complication rates higher in TOPS. The observation that dehiscence rates were higher in TOPS may reflect that surgeons more reliably track wound complications than the nurses contributing to NSQIP.
TOPS can be a key resource for plastic surgical outcomes research. However, careful attention must be paid to the population being studied and how one interprets the missing data problem. If accounting for the variation in patient populations between these datasets, the different strengths that each exhibits in complication reporting can be harnessed to improve plastic surgical outcomes.
Supplementary Material
Footnotes
Published online 27 May 2020.
Disclosure: The authors have no financial interest to declare in relation to the content of this article.
Related Digital Media are available in the full-text version of the article on www.PRSGlobalOpen.com.
REFERENCES
- 1.Molla Donaldson KL.Health Data in the Information Age: Use, Disclosure and Privacy. Institute of Medicine (U.S.). 1994Washington, D.C.: Committee on Regional Health Data Networks; [PubMed] [Google Scholar]
- 2.Blumenthal S.The use of clinical registries in the United States: a landscape survey. EGEMS (Wash DC). 2017;5:26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Shiloach M, Frencher SK, Jr, Steeger JE, et al. Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg. 2010;210:6–16. [DOI] [PubMed] [Google Scholar]
- 4.Hoque DME, Kumari V, Hoque M, et al. Impact of clinical registries on quality of patient care and clinical outcomes: a systematic review. PLoS One. 2017;12:e0183667. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Quality AfHRa. About AHRQ. https://www.ahrq.gov/cpi/about/index.html. Accessed August 21, 2019
- 6.Geubbels EL, Nagelkerke NJ, Mintjes-De Groot AJ, et al. Reduced risk of surgical site infections through surveillance in a network. Int J Qual Health Care. 2006;18:127–133. [DOI] [PubMed] [Google Scholar]
- 7.Arshi A, Rezzadeh K, Stavrakis AI, et al. Standardized hospital-based care programs improve geriatric hip fracture outcomes: an analysis of the ACS NSQIP targeted hip fracture series. J Orthop Trauma. 2019;33:e223–e228. [DOI] [PubMed] [Google Scholar]
- 8.Lundström M, Barry P, Brocato L, et al. European registry for quality improvement in cataract surgery. Int J Health Care Qual Assur. 2014;27:140–151. [DOI] [PubMed] [Google Scholar]
- 9.Stey AM, Russell MM, Ko CY, et al. Clinical registries and quality measurement in surgery: a systematic review. Surgery. 2015;157:381–395. [DOI] [PubMed] [Google Scholar]
- 10.Grover FL, Shroyer AL, Hammermeister K, et al. A decade’s experience with quality improvement in cardiac surgery using the veterans affairs and Society of Thoracic Surgeons national databases. Ann Surg. 2001;234:464–472; discussion 472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Khuri SF, Daley J, Henderson WG.The comparative assessment and improvement of quality of surgical care in the department of veterans affairs. Arch Surg. 2002;137:20–27. [DOI] [PubMed] [Google Scholar]
- 12.O’Neill AC, Murphy B, Bagher S, et al. Predicting complications in immediate alloplastic breast reconstruction: how useful is the American College of Surgeons National Surgical Quality Improvement Program surgical risk calculator? Plast Reconstr Surg. 2017;139:532–538. [DOI] [PubMed] [Google Scholar]
- 13.O’Neill AC, Bagher S, Barandun M, et al. Can the American College of Surgeons NSQIP surgical risk calculator identify patients at risk of complications following microsurgical breast reconstruction? J Plast Reconstr Aesthet Surg. 2016;69:1356–1362. [DOI] [PubMed] [Google Scholar]
- 14.Butler PD, Nelson JA, Fischer JP, et al. Racial and age disparities persist in immediate breast reconstruction: an updated analysis of 48,564 patients from the 2005 to 2011 American College of Surgeons National Surgery Quality Improvement Program data sets. Am J Surg. 2016;212:96–101. [DOI] [PubMed] [Google Scholar]
- 15.Kwok AC, Pannucci CJ, Agarwal JP.The American College of Surgeons National Surgical Quality Improvement Program flap failure data are inaccurate after 2010. Plast Reconstr Surg. 2016;138:570e–571e. [DOI] [PubMed] [Google Scholar]
- 16.Kwok AC, Agarwal JP.An analysis of free flap failure using the ACS NSQIP database. Does flap site and flap type matter? Microsurgery. 2017;37:531–538. [DOI] [PubMed] [Google Scholar]
- 17.Kwok AC, Goodwin IA, Ying J, et al. National trends and complication rates after bilateral mastectomy and immediate breast reconstruction from 2005 to 2012. Am J Surg. 2015;210:512–516. [DOI] [PubMed] [Google Scholar]
- 18.Kwok AC, Agarwal JP.Unplanned reoperations after microvascular free tissue transfer: an analysis of 2,244 patients using the American College of Surgeons National Surgical Quality Improvement Program database. Microsurgery. 2017;37:184–189. [DOI] [PubMed] [Google Scholar]
- 19.Vieira BL, Chow I, Sinno S, et al. Is there a limit? A risk assessment model of liposuction and lipoaspirate volume on complications in abdominoplasty. Plast Reconstr Surg. 2018;141:892–901. [DOI] [PubMed] [Google Scholar]
- 20.Khavanin N, Gutowski KA, Hume KM, et al. The use of patient registries in breast surgery: a comparison of the Tracking Operations and Outcomes for Plastic Surgeons and National Surgical Quality Improvement Program data sets. Ann Plast Surg. 2015;74:157–162. [DOI] [PubMed] [Google Scholar]
- 21.Surgeons ACO. About ACS NSQIP. https://www.facs.org/quality-programs/acs-nsqip/about. Accessed August 22, 2019
- 22.Surgeons ASOP. What is ASPS TOPS?. https://www.plasticsurgery.org/for-medical-professionals/registries/tracking-operations-and-outcomes-for-plastic-surgeons. Accessed August 21, 2019.
- 23.Lesko CR, Buchanan AL, Westreich D, et al. Generalizing study results: a potential outcomes perspective. Epidemiology. 2017;28:553–561. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Centers for Disease Control and Prevention (CDC) Use of a registry to improve acute stroke care—seven states, 2005–2009. MMWR Morb Mortal Wkly Rep. 2011;60:206–210. [PubMed] [Google Scholar]
- 25.Larsson S, Lawyer P, Garellick G, et al. Use of 13 disease registries in 5 countries demonstrates the potential to use outcome data to improve health care’s value. Health Aff (Millwood). 2012;31:220–227. [DOI] [PubMed] [Google Scholar]
- 26.McNeil JJ, Evans SM, Johnson NP, et al. Clinical-quality registries: their role in quality improvement. Med J Aust. 2010;192:244–245. [DOI] [PubMed] [Google Scholar]
- 27.Graves S.Benefits and limitations of registries. Paper presented at: Proceedings of the 54th Nordic Orthopaedic Federation Congress June 11-13, 2008 Amsterdam, The Netherlands. [Google Scholar]
- 28.Fischer JP, Wes AM, Serletti JM, et al. Complications in body contouring procedures: an analysis of 1797 patients from the 2005 to 2010 American College of Surgeons National Surgical Quality Improvement Program databases. Plast Reconstr Surg. 2013;132:1411–1420. [DOI] [PubMed] [Google Scholar]
- 29.Augustine HFM, Hu J, Najarali Z, et al. Scoping review of the National Surgical Quality Improvement Program in plastic surgery research. Plast Surg (Oakv). 2019;27:54–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Slump J, Ferguson PC, Wunder JS, et al. Can the ACS-NSQIP surgical risk calculator predict post-operative complications in patients undergoing flap reconstruction following soft tissue sarcoma resection? J Surg Oncol. 2016;114:570–575. [DOI] [PubMed] [Google Scholar]
- 31.Sellers MM, Merkow RP, Halverson A, et al. Validation of new readmission data in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg. 2013;216:420–427. [DOI] [PubMed] [Google Scholar]
- 32.American College of Surgeons. User Guide for the 2016 ACS NSQIP Participant Use File 2016. [Google Scholar]
- 33.Alderman AK, Collins ED, Streu R, et al. Benchmarking outcomes in plastic surgery: national complication rates for abdominoplasty and breast augmentation. Plast Reconstr Surg. 2009;124:2127–2133. [DOI] [PubMed] [Google Scholar]
- 34.Pisano SM, Ledoux PR, Nastala CL.Breast reconstruction in private practice. Semin Plast Surg. 2004;18:157–173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Chung CU, Wink JD, Nelson JA, et al. Surgical site infections after free flap breast reconstruction: an analysis of 2,899 patients from the ACS-NSQIP datasets. J Reconstr Microsurg. 2015;31:434–441. [DOI] [PubMed] [Google Scholar]
- 36.Nwaogu I, Yan Y, Margenthaler JA, et al. Venous thromboembolism after breast reconstruction in patients undergoing breast surgery: an American College of Surgeons NSQIP analysis. J Am Coll Surg. 2015;220:886–893. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.