Optimizing Risk-Adjusted Outcome Measures: A Moving Target

The optimal risk-adjustment tool to measure quality of surgical care would control for patient comorbidities, complexity of operations, physician and hospital case mix, and specialty-specific conditions. This ideal tool would help to produce generalizable and disease-specific risk-adjusted measurements of mortality, surgical site infection rates, and other surgical outcomes. Unfortunately no such risk adjustment tool exists. However, the American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) is in a position to develop this tool in its continuous efforts to provide risk-adjusted performance data that participating sites can use to improve the quality of care delivered to surgical patients [1].

The study by Goldberg and colleagues [2] demonstrates specific limitations in the ACS-NSQIP risk-adjusted outcome measures, which may guide future improvements to the risk adjustment tool. Many of us who utilize the NQSIP participant use data file for health services research or quality improvement efforts may already be aware of some limitations but have yet to see this type of global assessment published in a simple but informative way. For those who are new to working with the NSQIP participant use file, this article demonstrates that even this robust tool for measuring surgical outcomes has specific limitations that should be heeded when designing a study. As NSQIP is progressively improved, the methods for risk adjustment will need to be revamped. The greater message of this study, beyond NSQIP, is that the process of risk adjustment in measuring healthcare quality is a moving target. The NSQIP and other healthcare quality improvement programs face significant challenges in optimizing risk-adjusted outcome measures. These challenges urgently need to be confronted as pay-for-performance policies, based on risk-adjusted quality measurements, are increasingly implemented.

Participation in the NSQIP will likely fulfill reporting requirements for the Physician Quality Reporting System (PQRS) mandated by the Affordable Care Act [3]. Centers for Medicare & Medicaid Services (CMS) will levy a 2% reduction in reimbursement to healthcare providers who do not report to PQRS by 2016, either through Medicare claims, a CMS approved electronic health record, a qualified registry or measures group, or the group practice reporting option [4]. The National Quality Forum, which is contracted by the U.S. Department of Health and Human Services to develop healthcare quality and efficiency measures, recently endorsed the NSQIP surgical site infection and urinary tract infection quality measures [5], which could potentially be added to the list of PQRS quality measures. Therefore NSQIP risk-adjusted outcomes will be of critical importance to many surgeons nationwide with regard to reimbursement and reporting to regulatory agencies.

Auditors and quality improvement consultants have frequently heard the assertion, “Outcomes at this hospital are worse because we take care of… sicker patients… an elderly population… more complex cases.” With mandatory reporting of outcomes on the horizon, physicians and hospitals are hoping that risk-adjustment will help to accurately reflect their quality of care in comparison to others’. Interestingly, the National Veterans Affairs Surgical Risk Study, a predecessor to NSQIP, found that risk-adjustment only modestly affected the ultimate rankings of hospitals [6]. This suggests that either the risk-adjustment tool did not adequately control for significant differences that affected outcomes, or that risk adjustment did not make a real difference when measuring quality. The study by Goldberg and colleagues [2] suggests that a risk adjustment tool which better controls for case mix and age of the patient population would significantly affect the ultimate measurement of quality in the 2008 NSQIP dataset. Unfortunately statistical tests were not performed in the study to determine if differences in Observed/Expected ratios were statistically significant with regard to age and case mix. This is both a limitation of the study and an opportunity for the NSQIP to explore these statistical relationships using the full dataset, rather than the participant use data file.

Several studies have described limitations in risk-adjusted outcome measurement using NSQIP data [7–11]. Despite these limitations, there is evidence that the risk adjustment model utilized by the NSQIP outperforms other risk adjustment models when determining expected values for mortality [12]. One study demonstrated that the NSQIP model more accurately predicted mortality compared to the Charlson Comorbidity Index and the DxCG model that utilizes Diagnostic Cost Groups [13]. In the 2008 NSQIP participant use data file, probability of mortality was calculated using logistic regression analysis with patient's preoperative characteristics (approximately 65 variables) as the potential predictors. More recent studies have demonstrated that as few as 3 to 5 variables can instead be used to predict mortality just as reliably [14–16]. Therefore, it could be argued that the NSQIP uses more than enough data to risk-adjust mortality measurements.

Risk adjustment of disease-specific morbidity and specialty-specific complications is much more complex. The National Veterans Affairs Surgical Risk Study demonstrated little to no correlation between the rank order of hospitals by risk-adjusted morbidity versus risk-adjusted mortality [6]. This suggests that distinct methods may be required to measure non-fatal disease processes or complications as opposed to mortality rates. Disease-specific and surgical specialty-specific measures may be better means of producing risk-adjusted morbidity measurements [17]. A recent study performed using NSQIP data found that a risk adjustment model incorporating a Current Procedural Terminology (CPT) codes risk score more accurately predicted morbidities [9]. Therefore grouping outcomes by procedure or CPT code can provide more accurate assessments of the quality of care in specific surgery types. Furthermore, CPT codes should be included in the assessment of reoperation as a quality measure. The study by Goldberg and colleagues [2] as well as studies performed by breast surgeons [7], thyroid surgeons [10], and other surgical oncologists [8, 11] report that 30-day reoperation can represent management of a complication (e.g. bleeding), preventable re-excision of cancer, or non-preventable completion of the cancer operation. As McCahill and colleagues point out, preventable re-excision of tumor may be considered in future quality measures for breast surgery [18] and potentially other cancer operations. However, a non-specific unplanned reoperation variable may not accurately reflect post-operative morbidity. Involvement of surgical subspecialists in the development of disease-specific and surgical specialty-specific quality measures is essential in order to implement a fair system of reporting surgical outcomes and policies to improve those outcomes.

The NSQIP offers a procedure-targeted participation option that includes measurement of procedure-specific variables for general, thoracic, vascular, gynecologic, urologic, orthopedic, neurosurgery, otolaryngology, plastic and reconstructive surgeries [1]. Procedures included in the general surgery module are pancreatectomy, hepatectomy, esophagectmy, colectomy, proctectomy, appendectomy, ventral hernia repair, bariatric surgery, and thyroidectomy. Some sub-specialty groups have made significant efforts to develop their own quality reporting systems that have been incorporated into the PQRS list of measures. The American Society of Breast Surgeons, American Society of Clinical Oncology, National Comprehensive Cancer Network, Society of Thoracic Surgeons, and Society of Vascular Surgeons contributed to the development of disease-specific surgical quality measures in PQRS [4]. As national policies increasingly regulate the practice of surgery, it is important that surgeons increasingly become involved in the development of surgical outcome measures and specialty-specific optimization of risk adjustment. The ACS-NSQIP is in an ideal position to foster involvement of surgeons in this process of optimizing risk-adjusted outcome measures.