Risk-Adjusted Pathologic Margin Positivity Rate As a Quality Indicator in Rectal Cancer Surgery

Nader N Massarweh; Chung-Yuan Hu; Y Nancy You; Brian K Bednarski; Miguel A Rodriguez-Bigas; John M Skibber; Scott B Cantor; Janice N Cormier; Barry W Feig; George J Chang

doi:10.1200/JCO.2014.55.5334

. 2014 Aug 4;32(27):2967–2974. doi: 10.1200/JCO.2014.55.5334

Risk-Adjusted Pathologic Margin Positivity Rate As a Quality Indicator in Rectal Cancer Surgery

Nader N Massarweh ¹, Chung-Yuan Hu ¹, Y Nancy You ¹, Brian K Bednarski ¹, Miguel A Rodriguez-Bigas ¹, John M Skibber ¹, Scott B Cantor ¹, Janice N Cormier ¹, Barry W Feig ¹, George J Chang ^1,^✉

PMCID: PMC4162495 PMID: 25092785

Abstract

Purpose

Margin positivity after rectal cancer resection is associated with poorer outcomes. We previously developed an instrument for calculating hospital risk-adjusted margin positivity rate (RAMP) that allows identification of performance-based outliers and may represent a rectal cancer surgery quality metric.

Methods

This was an observational cohort study of patients with rectal cancer within the National Cancer Data Base (2003 to 2005). Hospital performance was categorized as low outlier (better than expected), high outlier (worse than expected), or non-RAMP outlier using standard observed-to-expected methodology. The association between outlier status and overall risk of death at 5 years was evaluated using Cox shared frailty modeling.

Results

Among 32,354 patients with cancer (mean age, 63.8 ± 13.2 years; 56.7% male; 87.3% white) treated at 1,349 hospitals (4.9% high outlier, 0.7% low outlier), 5.6% of patients were treated at high outliers and 3.0% were treated at low outliers. Various structural (academic status and volume), process (pathologic nodal evaluation and neoadjuvant radiation therapy use), and outcome (sphincter preservation, readmission, and 30-day postoperative mortality) measures were significantly associated with outlier status. Five-year overall survival was better at low outliers (79.9%) compared with high outliers (64.9%) and nonoutliers (68.9%; log-rank test, P < .001). Risk of death was lower at low outliers compared with high outliers (hazard ratio [HR], 0.61; 95% CI, 0.50 to 0.75) and nonoutliers (HR, 0.69; 95% CI, 0.57 to 0.83). Risk of death was higher at high outliers compared with nonoutliers (HR, 1.12; 95% CI, 1.03 to 1.23).

Conclusion

Hospital RAMP outlier status is a rectal cancer surgery composite metric that reliably captures hospital quality across all levels of care and could be integrated into existing quality improvement initiatives for hospital performance.

INTRODUCTION

Colorectal cancer is among the top incident cancers and leading causes of cancer-related mortality in the United States, with rectal cancer responsible for roughly one third of these deaths.¹ Although rectal cancer treatment has evolved (eg, total mesorectal excision, neoadjuvant radiation therapy), circumferential margin positivity remains a highly variable outcome and key factor associated with patient outcome.^2,3 A negative surgical margin is critical because adjuvant therapies are an ineffective means of compensating for margin positivity.^4,5 In the United States, ongoing efforts to develop a multidisciplinary quality improvement program are focusing on various aspects of rectal cancer care, one of which is circumferential radial margin reporting.⁶

Identifying valid quality metrics for complex cancer surgery, as well as methods to measure and report them, is important in the ongoing dialog about improving health care quality and lowering costs. As yet, the optimal method to define and measure surgical quality remains elusive.⁷ Much of the difficulty is rooted in the complexity of the current health care environment and because opportunities for quality improvement can occur in any of three dimensions—structure, process, or outcome.⁸ Although metrics based on health care structure and patient outcome are generally easily measured and reported, each has distinct disadvantages. The former are typically immutable, and the mechanism by which structure impacts outcomes is not always clearly defined; therefore, the value and policy implications of a structural metric as a proxy for quality may not be readily apparent.^9–11 For outcome measures, identifying granular data allowing adequate risk adjustment can be a challenge, postoperative outcomes may be less actionable than surgical processes, issues with small sample size and low event rates may impact reliability, and any given outcome may not accurately reflect the quality of care a patient actually received.^7,12,13 By comparison, process-based metrics are readily modifiable and offer hospitals and providers potentially definable targets for improvement. However, procedure-specific perioperative process metrics have not been well defined.

For oncologic care, a major hurdle is the often complex, multidisciplinary nature of treatment and evolving practice standards. Rectal cancer offers a prime example; although the primary treatment is surgical resection, the spectrum of care often involves multimodality therapy with neoadjuvant or adjuvant chemotherapy and/or radiation therapy. Another key challenge is defining measures that capture the full range of factors involved. For patients with rectal cancer, patient-, tumor-, and treatment-related factors all impact a given patient's risk for a positive surgical margin. Finding ways to account for these issues is critical to defining an adequate surgical quality metric.

Our group developed a tool for identifying performance-based outlier hospitals based on patient and tumor risk-adjusted margin positivity rate (RAMP) after rectal cancer surgery.¹⁴ The purpose of this tool is to adjust for relevant patient and tumor factors beyond the control of the treating hospital, thereby allowing high-performing hospitals to highlight best practices and underperformers to better understand whether they might benefit from quality improvement efforts allowing them to critically appraise their multidisciplinary care processes. RAMP outlier status captures many of the elements impacting clinical decision making, therapeutic recommendations, and a patient's risk for margin positivity and long-term outcome. As such, the goal of this study is to evaluate the potential value of hospital RAMP outlier status as a rectal cancer surgery quality indicator. Our hypothesis is that outlier status is associated with patient survival (considered the most important quality indicator in surgical and cancer care) and other health care structural factors and process of care measures correlated with quality rectal cancer care.

METHODS

Data

The National Cancer Data Base (NCDB) is a prospective, hospital-based cancer registry collecting and reporting patient data on more than 70% of cancers diagnosed in the United States. Since the inception of this joint project of the American College of Surgeons Commission on Cancer (CoC) and the American Cancer Society, data have been collected on approximately 25 million patients with cancer treated at over 1,500 participating centers. Comprehensive discussions of the NCDB have been previously published.^15–17 The data did not include any patient identifiers, and therefore, this project was considered exempt by The University of Texas MD Anderson Cancer Center Institutional Review Board.

Study Patients

Figure 1 is a flow diagram of study criteria and analytic group definitions. Our cohort was restricted to patients diagnosed after 2003 because comorbidity data were added beginning that year. Patient diagnosed after 2005 were excluded because complete, long-term follow-up survival data are currently only available for patients diagnosed through 2005. Patients with prior cancer diagnoses were excluded. Patients not treated at the reporting institution were excluded to increase the accuracy of treatment ascertainment.

Fig 1. — Flow diagram of study design. AJCC, American Joint Committee on Cancer.

Variables

Demographic, clinical, and tumor data are provided in the NCDB. Indicators of income and education are provided (based on area of residence derived from Census 2000 data). A Charlson-Deyo comorbidity index is also provided.¹⁸ Pathologic (as opposed to clinical) staging data were used for the following reasons: it is most relevant to the evaluation of a positive margin; clinical staging is heavily predicated on choice of preoperative imaging modality (endoscopic ultrasound v pelvic magnetic resonance imaging) and the conduct and accuracy of interpretation; and pathologic staging accounts for the effect of any neoadjuvant therapy that may have been used. Surgical volume for each facility was calculated using average annual number of rectal resections performed for cancer stratified into quartiles.

Analysis

Descriptive statistics were used to evaluate categorical and continuous variable distributions. A nonparametric test for trend was used to evaluate changes in rate across outlier categories. Using a previously developed and internally validated nomogram,¹⁴ we created the following three analytic groups based on observed-to-expected (O/E) ratios for margin positivity: high outliers (worse than expected performers), low outliers (better than expected performers), and nonoutliers. Identification of significant O/E outliers was based on the exact binomial function simultaneously taking into account hospital volume: P (k out of n) = $\frac{n |}{k | (n - k) |}$ (p^k)(1 − p)^{n − k}; where p denotes the model-based estimated probability of being margin positive for a specific hospital; n denotes total volume of patients treated in a specific hospital; and k denotes the actually observed number of margin-positive patients in a specific hospital.^19,20

The primary outcome was 5-year overall survival. Multivariable Cox-shared frailty modeling (assuming a gamma distribution) was used to evaluate the association between risk of death and hospital outlier status.²¹ Multilevel modeling was used because the data were considered correlated (patients treated at the same hospital were potentially similar in terms of the care they received and their eventual outcome). Models were created using a nonparsimonius approach. The proportional hazards assumption was assessed graphically. In addition, risk of death was evaluated among the subgroup with non-T4 tumors (because stage of disease was the dominant predictor in the nomogram, with T4 tumors being the driving factor [n = 30,732]). Sensitivity analyses were also performed among patients who had a margin-negative resection (n = 30,629), who received neoadjuvant radiation therapy (n = 7,807), and who had ≥ 12 lymph nodes pathologically evaluated (n = 15,007), and excluding patients treated at hospitals reporting observed margin positivity rates of zero (n = 24,822). In addition, the data were modeled using the following five-tier outlier classification: high outlier with O/E more than 1 and P < .05; at-risk high outlier with O/E more than 1 and P > .05 but < .1; low nonoutlier with O/E less than 1 and P > .05 but < .1; low outlier with O/E less than 1 and P < .05; and nonoutlier with O/E more than 1 and P > .1 and less than 1 and P > .1.

Overall, 23.8% of patients were missing at least one covariate data point (22.5% high outlier; 23.7% nonoutlier; 30.9% low outlier; P < .001). Five-year survival was not significantly different among patients with and without missing data in either the low (81.9% missing v 79.1% nonmissing; log-rank, P = .26) or high (68.1% missing v 63.9% nonmissing; log-rank, P = .27) outlier groups. There was a significant survival difference in 5-year survival among patients with and without missing data in the nonoutlier group (72.5% missing v 67.9% nonmissing; log-rank, P < .001). Variation based on missing data was deemed not clinically meaningful. Nonetheless, to ensure model results were robust to varying assumptions about missing data, we performed a case-complete analysis followed by an imputed analysis using five sets of data obtained through multiple imputation by chained equations.^22,23 Point estimates and inference from both analyses were similar; therefore, imputed results are reported. All analyses were performed using STATA 11.1 (StatCorp, College Station, TX). Statistical comparisons were two-sided and considered significant at the P < .05 level.

RESULTS

A total of 32,354 patients with rectal cancer were included. Mean age was 63.8 ± 13.2 years, 56.7% were male, and 87.3% were white. The overall observed rate of margin positivity was 5.3% (17.1% for high outliers, 4.8% for nonoutliers, and 0.8% for low outliers; P < .001). Table 1 lists demographic and clinical characteristics stratified by hospital outlier status. Among 1,349 treating hospitals included in the analysis, 66 (4.9%) were high RAMP outliers treating 5.6% of the patients and nine (0.7%) were low outliers treating 3.0% of the patients.

Table 1.

Patient Demographic and Clinical Characteristics Stratified by Hospital RAMP Outlier Status

Demographic or Clinical Characteristic	Hospital RAMP Outlier Status (% of patients)			P
Demographic or Clinical Characteristic	High Outlier (n = 1,797)	Nonoutlier (n = 29,585)	Low Outlier (n = 972)	P
No. of hospitals	66	1,274	9
Age, years				< .001
Mean	64.0	64.0	61.1
SD	13.2	13.2	13.4
Age at diagnosis, years				< .001
18-49	14.5	14.4	21.1
50-64	35.1	35.9	36.2
65-74	26.2	25.1	25.6
75-90	24.2	24.6	17.1
Male sex	57.0	56.6	57.0	.93
Race				.02
White	86.9	87.3	87.8
Black	8.9	7.4	6.3
Other	4.2	5.2	6.0
Insurance status				< .001
Uninsured	4.1	2.8	3.8
Private	11.9	13.5	10.1
Medicaid	3.7	3.7	3.9
Medicare	46.0	44.9	33.4
Other	31.2	33.2	34.5
Missing	3.1	1.9	14.3
Income^*				< .001
≥ $46,000	31.2	36.7	48.5
Missing	6.4	5.5	1.5
Education^*				< .001
≥ 29%	19.2	16.3	13.2
Missing	6.4	5.5	1.5
Rurality				< .001
Metropolitan	76.4	76.0	84.3
Suburban	16.6	15.6	13.2
Rural	1.3	2.3	1.6
Missing	5.7	6.2	0.9
Tumor location				< .001
Rectosigmoid	38.2	37.9	31.7
Rectum	61.8	62.1	68.3
AJCC stage, sixth edition				.26
I	29.0	31.3	33.7
IIA	27.2	26.7	25.1
IIB	2.6	2.4	2.3
IIIA	7.1	7.2	8.1
IIIB	17.4	17.4	16.8
IIIC	16.8	14.9	14.0
Tumor size, mm				.02
≤ 10	3.4	3.9	4.7
11-20	9.3	10.0	10.3
> 20	73.0	71.3	66.7
Missing	14.2	14.8	18.3
Histology				.44
Adenocarcinoma	91.8	91.9	90.5
Signet ring	0.7	0.7	0.6
Mucinous	7.5	7.3	8.8
Tumor grade				< .001
Well/moderately differentiated	81.1	80.9	72.0
Poorly/undifferentiated	13.7	13.7	22.3
Missing	5.2	5.4	5.7
Comorbidity score				.006
0	81.6	78.4	80.2
1	14.9	17.5	15.0
≥ 2	3.5	4.1	4.7

Open in a new tab

NOTE. Column percentages may not add up to 100% because of rounding.

Abbreviations: RAMP, risk-adjusted margin positivity rate; SD, standard deviation.

Based on 2000 census data. For income, the percentage of patients whose area of residence (based on year 2000 census data) had a median household income ≥ $46,000 is presented. For education, the percentage of patients whose area of residence (based on year 2000 Census data) had ≥ 29% of adults who did not attain a high school education is presented.

We evaluated the association between outlier status and several other structure, process, and outcome factors (Fig 2). Overall, 29.8% of patients were treated at academic institutions (26.0% high outlier, 28.2% nonoutlier, and 84.3% low outlier; trend test, P < .001) and 53.4% received treatment at high-volume centers (51.8% high outlier, 52.1% nonoutlier, and 96.7% low outlier; trend test, P < .001). Within the cohort, 46.8% of patients had ≥ 12 lymph nodes evaluated, and 25.3% received neoadjuvant radiation therapy. RAMP outlier status was associated with higher proportions of patients with ≥ 12 nodes evaluated (46.4% high outlier, 46.3% nonoutlier, and 60.8% low outlier; trend test, P < .001) and who received neoadjuvant radiation therapy (23.5% high outlier, 23.9% nonoutlier, and 34.6% low outlier; trend test, P < .001). Overall rate of sphincter-preserving procedures (excluding patients with rectosigmoid tumors) was 71.0%. Postoperative 30-day readmission and mortality occurred for 9.1% and 1.8% of patients, respectively. There were significant trends favoring higher rates of sphincter preservation (69.4% high outlier, 70.9% nonoutlier, and 75.0% low outlier; trend test, P = .02), lower readmission rates (11.0% high outlier, 9.7% nonoutlier, and 8.0% low outlier; trend test, P = .01), and lower 30-day mortality rates (2.2% high outlier, 1.8% nonoutlier, and 0.7% low outlier; trend test, P = .007) at low-outlier hospitals.

Unadjusted 5-year overall survival (Fig 3A) was significantly different among the various outlier categories (low outlier, 79.9%; nonoutlier, 68.9%; high outlier, 64.9%; log-rank, P < .001). A significant difference in 5-year overall survival persisted when the cohort was restricted to patients with non-T4 tumors (Fig 3B; low outlier, 81.1%; nonoutlier, 70.4%; high outlier, 66.9%; log-rank, P < .001). After multilevel modeling (Table 2), patients treated at low-outlier hospitals had a 30% (hazard ratio [HR], 0.69; 95% CI, 0.57 to 0.83) and 40% (HR, 0.61; 95% CI, 0.50 to 0.75) lower risk of death at 5 years compared with patients treated at nonoutlier and high-outlier hospitals, respectively. Patients treated at high-outlier hospitals had a significantly higher risk of death compared with those treated at nonoutlier hospitals (HR, 1.12; 95% CI, 1.03 to 1.23). When the cohort was restricted to patients with non-T4 tumors, there were similar statistically significant differences in the risk of death (∼40% decrease [low v high outlier], ∼30% decrease [low v nonoutlier], and ∼10% increase [high v nonoutlier]).

Table 2.

Association Between Hospital RAMP Outlier Status and 5-Year Overall Survival

Outlier Status	Overall Cohort		Subgroup With Non-T4 Tumors
Outlier Status	Hazard Ratio	95% CI	Hazard Ratio	95% CI
Unadjusted
High outlier	1.16	1.05 to 1.28	1.15	1.04 to 1.27
Nonoutlier	1.00		1.00
Low outlier	0.69	0.56 to 0.84	0.69	0.56 to 0.84
Low v high outlier (ref)	0.59	0.47 to 0.73	0.60	0.48 to 0.75
Adjusted for demographics^*
High outlier	1.14	1.04 to 1.25	1.12	1.02 to 1.24
Nonoutlier	1.00		1.00
Low outlier	0.72	0.60 to 0.86	0.73	0.60 to 0.87
Low v high outlier (ref)	0.63	0.51 to 0.76	0.64	0.52 to 0.79
Adjusted for demographics and clinical characteristics^†
High outlier	1.12	1.03 to 1.23	1.11	1.01 to 1.23
Nonoutlier	1.00		1.00
Low outlier	0.69	0.57 to 0.83	0.70	0.58 to 0.84
Low v high outlier (ref)	0.61	0.50 to 0.75	0.62	0.50 to 0.77

Open in a new tab

Abbreviations: RAMP, risk-adjusted margin positivity rate; ref, reference.

Inclues age, sex, race, year of diagnosis, income, education, insurance, and rurality.

^†

Includes the above plus comorbidity, stage of disease, tumor size, tumor histology, tumor grade, and tumor location (rectum v rectosigmoid).

Additional sensitivity analyses were performed to evaluate the robustness of the association between RAMP outlier status and overall survival (Table 3). The additional analyses were restricted to separate subgroups of patients who had a margin-negative resection, received neoadjuvant radiation, and had ≥ 12 lymph nodes pathologically evaluated, and after excluding hospitals reporting no observed margin-positive patients. In addition, the data were modeled using an alternative, five-level outlier classification scheme. In general, the magnitude and direction of the effect in these sensitivity analyses were consistent with our primary analysis.

Table 3.

Frailty Model Sensitivity Analyses

Factor and Outlier Status	Hazard Ratio	95% CI
Margin-negative patients
High outlier	1.05	0.95 to 1.17
Nonoutlier	1.00
Low outlier	0.71	0.58 to 0.86
Low v high outlier (ref)	0.67	0.54 to 0.83
Neoadjuvant XRT^*
High outlier	1.23	0.94 to 1.61
Nonoutlier	1.00
Low outlier	0.67	0.39 to 1.14
Low v high outlier (ref)	0.55	0.32 to 0.94
≥ 12 lymph nodes pathologically evaluated^*
High outlier	1.13	0.99 to 1.28
Nonoutlier	1.00
Low outlier	0.73	0.58 to 0.91
Low v high outlier (ref)	0.64	0.50 to 0.83
Excluding hospitals with no margin-positive patients^*
High outlier	1.12	1.02 to 1.23
Nonoutlier	1.00
Low outlier	0.59	0.47 to 0.74
Low v high outlier (ref)	0.52	0.41 to 0.67
Alternate RAMP stratification^†
High outlier	1.14	1.04 to 1.25
At-risk high outlier	1.02	0.93 to 1.12
Nonoutlier	1.00
Low nonoutlier	0.78	0.65 to 0.94
Low outlier	0.68	0.57 to 0.82

Open in a new tab

Abbreviations: RAMP, risk-adjusted margin positivity rate; ref, reference; XRT, radiotherapy.

Models were restricted to patients who received neoadjuvant XRT, who had ≥ 12 lymph nodes evaluated pathologically, and who were treated at hospitals reporting no observed margin-positive patients.

^†

Alternate stratification as follows: high outlier: observed-to-expected ratio (O/E) > 1 and P < .05; at-risk high outlier: O/E > 1 and P > .05 but < .1; low nonoutlier: O/E < 1 and P > .05 but < .1; and low outlier: O/E < 1 and P < .05.

DISCUSSION

Although contemporary health policy and quality improvement initiatives in surgery remain predominantly focused on metrics based on health care structure and outcome,^16,24 finding appropriate process measures is desirable because they best represent the care a patient actually received and may be the most actionable. Although various surgical process metrics have been put forth, their value and association with the quality of delivered care and patient outcomes are questionable.^25–27 In rectal cancer surgery, circumferential margin positivity is associated with local recurrence and patient survival.²⁸ Proper patient selection for and utilization of neoadjuvant radiation impacts the risk of pathologic margin positivity and, subsequently, local recurrence.^29,30 Therefore, appropriate multidisciplinary care, taking into account all relevant patient and tumor factors that might impact a given patient's likelihood of having a positive circumferential margin, is crucial to treatment sequencing and ultimately achieving a margin-negative resection. Herein, we propose hospital performance in terms of patient and tumor risk-adjusted pathologic margin positivity after rectal cancer resection as a valid rectal cancer surgery quality indicator.¹⁴ In our current work, meaningful health care structure (hospital volume and academic affiliation), process of care (pathologic nodal evaluation and neoadjuvant radiation utilization), and outcome (sphincter preservation, readmission, and overall survival) metrics were all associated with hospital outlier status, suggesting that RAMP outlier status may be a composite metric for rectal cancer surgery that reliably captures hospital-level quality across all levels of patient care.

Among the challenges facing health care quality improvement initiatives are identification of reliable quality metrics, understanding how best to report quality data, and the timeliness with which data are presented to stakeholders. Desirable features of a quality metric have been previously described and include high reliability and validity, low cost of ascertainment, actionable, and well-defined objective.³¹ Our findings suggest that hospital RAMP outlier status fulfills these criteria and may constitute a metric with high fidelity and tight linkage to meaningful patient outcomes. Compared with mortality, a notable advantage is that it is measured relatively early in the spectrum of surgical care, offering providers opportunities to evaluate their performance and allowing outliers the chance to target their internal processes that would benefit future patients receiving treatment at those centers.

Another important component of any quality improvement effort is gaining buy-in from relevant stakeholders. Much of this depends on the culture fostered within a given institution or system. For example, top-performing institutions actively and consistently stress opportunities for improvement, whereas others either consider the status quo or minimal/gradual gains in performance acceptable or overestimate their level of performance.^32,33 This disparity may account for some of the challenges in implementing hospital-based quality improvement efforts and performance-based reporting initiatives as a means to improve the quality of care at the individual patient level. However, the American College of Surgeons National Surgical Quality Improvement Program, Washington state's Surgical Care and Outcomes Assessment Program, and others provide existing examples of how reporting institution-level data not only improves adherence to surgical processes at the provider level, but also impacts patient outcomes and health care costs.^20,34 Using rectal cancer resection as an archetype, although the surgical margin is largely determined by the quality of the operation performed by the individual surgeon, there are often regional and/or institutional biases inherent to the multidisciplinary care provided that can impact a given patient's risk for a positive surgical margin, even after adjusting for relevant patient and tumor factors. We believe monitoring and reporting RAMP outlier status could provide institutions with meaningful, nonpunitive performance-based feedback, thereby allowing them to examine how their providers deliver care to patients with rectal cancer and offering a benchmark for improvement.

Programs around the world that have focused on improving the quality of rectal cancer surgery with auditing and feedback have yielded improved outcomes.³⁵ Yet, in part due to system-level differences between the United States and other countries with public health care, no such program has been successfully implemented in the United States. Measuring RAMP outlier status could be integrated into nearly any contemporary quality improvement model.¹² As an added advantage, the NCDB offers a national platform for implementation with pre-existing infrastructure to measure, report, and disseminate data to CoC-accredited and NCDB participating hospitals through two Web-based hospital quality improvement programs. The Rapid Quality Reporting System offers feedback regarding adherence to quality metrics.³⁶ The Cancer Program Practice Profile Report provides comparative performance data for various standard cancer care practices.³⁷ RAMP outlier status could be integrated into both of these online programs and offer hospitals comparative, real-time data.

This work has several important limitations. The instrument used has only been validated in NCDB. Ideally, this tool would be applicable to other existing registry or administrative data sets. Unfortunately, although the component variables are commonly measured in other data sets, to our knowledge, only NCDB contains all the requisite variables. We were unable to measure and therefore adjust for receipt of postoperative chemotherapy because only one time point for initiation of a given therapy is provided (eg, patients who may have received neoadjuvant and adjuvant chemotherapy would only have the date of preoperative initiation reported). Our data were restricted to CoC-accredited hospitals. Therefore, the generalizability of our findings to non-CoC hospitals is unclear.^38,39 Data on provider specialty/training may have helped in understanding the impact of provider-level factors on hospital outlier status; however, provider data are not available. Although we would have liked to understand the impact clinical staging may have had on recommended/administered preoperative therapies by the various outlier institutions, we were only able to reliably evaluate pathologic tumor stage. Our cohort included a proportion of missing covariate data. However, survival was actually slightly better among patients with missing data in all three outlier categories, differences in survival were either statistically or clinically nonsignificant, and the results were robust to varying methods of handling missing data—all suggesting that missing data likely had a minimal impact on the study results. Finally, there are potential unintended consequences of such quality improvement efforts, such as broader use of neoadjuvant radiation among patients for whom it may not be indicated, patients at higher risk for margin-positive resection being referred away from local institutions based on outlier status, or patients electively seeking care at more specialized centers.

Our work demonstrates that hospital RAMP outlier status represents a valid and useful composite quality indicator for rectal cancer surgery. Furthermore, the strong association between RAMP outlier status and numerous other structural, process, and outcome factors and the pre-existing platform for dissemination and utilization of this tool make a compelling case for modeling future surgical quality metrics on this instrument. As put forth by Birkmeyer et al⁷ “Quality in health care can be described as ‘doing the right things right.’” Although simple in principle, the actual practice of finding the right things to measure, allowing stakeholders to properly gauge surgical quality, has proven difficult. As quality evaluations in health care continue to evolve, stakeholders must constantly ensure that measurement and reporting are not thought of as outcomes in and of themselves. Rather, what is measured should provide feedback and positively impact the care a patient receives.

Footnotes

See accompanying editorial on page 2938

Supported by National Institutes of Health/National Cancer Institute Grants No. K07-CA133187 (G.J.C.) and CA16672 (The University of Texas MD Anderson Cancer Center Support Grant).

Presented in part at the 2014 Academic Surgical Congress, San Diego, CA, February 4-6, 2014.

The data used in this study are derived from a deidentified National Cancer Data Base file. The American College of Surgeons and the Commission on Cancer have not verified and are not responsible for the analytic or statistical methodology used or the conclusions drawn from these data.

Authors' disclosures of potential conflicts of interest and author contributions are found at the end of this article.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The author(s) indicated no potential conflicts of interest.

AUTHOR CONTRIBUTIONS

Conception and design: Nader N. Massarweh, George J. Chang

Financial support: George J. Chang

Administrative support: George J. Chang

Provision of study materials or patients: Janice N. Cormier, George J. Chang

Collection and assembly of data: Nader N. Massarweh, Chung-Yuan Hu, Janice N. Cormier, George J. Chang

Data analysis and interpretation: All authors

Manuscript writing: All authors

Final approval of manuscript: All authors

REFERENCES

1.Siegel R, Naishadham D, Jemal A. Cancer statistics, 2012. CA Cancer J Clin. 2012;62:10–29. doi: 10.3322/caac.20138. [DOI] [PubMed] [Google Scholar]
2.Wibe A, Rendedal PR, Svensson E, et al. Prognostic significance of the circumferential resection margin following total mesorectal excision for rectal cancer. Br J Surg. 2002;89:327–334. doi: 10.1046/j.0007-1323.2001.02024.x. [DOI] [PubMed] [Google Scholar]
3.Adam IJ, Mohamdee MO, Martin IG, et al. Role of circumferential margin involvement in the local recurrence of rectal cancer. Lancet. 1994;344:707–711. doi: 10.1016/s0140-6736(94)92206-3. [DOI] [PubMed] [Google Scholar]
4.Marijnen CA, Nagtegaal ID, Kapiteijn E, et al. Radiotherapy does not compensate for positive resection margins in rectal cancer patients: Report of a multicenter randomized trial. Int J Radiat Oncol Biol Phys. 2003;55:1311–1320. doi: 10.1016/s0360-3016(02)04291-8. [DOI] [PubMed] [Google Scholar]
5.Quirke P, Steele R, Monson J, et al. Effect of the plane of surgery achieved on local recurrence in patients with operable rectal cancer: A prospective study using data from the MRC CR07 and NCIC-CTG CO16 randomised clinical trial. Lancet. 2009;373:821–828. doi: 10.1016/S0140-6736(09)60485-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Fleshman JW. Multidisciplinary treatment of rectal cancer: The way of the future. JAMA Surg. 2013;148:778. doi: 10.1001/jamasurg.2013.2151. [DOI] [PubMed] [Google Scholar]
7.Birkmeyer JD, Dimick JB, Birkmeyer NJ. Measuring the quality of surgical care: Structure, process, or outcomes? J Am Coll Surg. 2004;198:626–632. doi: 10.1016/j.jamcollsurg.2003.11.017. [DOI] [PubMed] [Google Scholar]
8.Donabedian A. The quality of care: How can it be assessed? JAMA. 1988;260:1743–1748. doi: 10.1001/jama.260.12.1743. [DOI] [PubMed] [Google Scholar]
9.Christian CK, Gustafson ML, Betensky RA, et al. The volume-outcome relationship: Don't believe everything you see. World J Surg. 2005;29:1241–1244. doi: 10.1007/s00268-005-7993-8. [DOI] [PubMed] [Google Scholar]
10.Daley J. Invited commentary: Quality of care and the volume-outcome relationship—What's next for surgery? Surgery. 2002;131:16–18. doi: 10.1067/msy.2002.120237. [DOI] [PubMed] [Google Scholar]
11.Sheikh K. Reliability of provider volume and outcome associations for healthcare policy. Med Care. 2003;41:1111–1117. doi: 10.1097/01.MLR.0000088085.61714.AE. [DOI] [PubMed] [Google Scholar]
12.Birkmeyer NJ, Birkmeyer JD. Strategies for improving surgical quality: Should payers reward excellence or effort? N Engl J Med. 2006;354:864–870. doi: 10.1056/NEJMsb053364. [DOI] [PubMed] [Google Scholar]
13.Dimick JB, Welch HG, Birkmeyer JD. Surgical mortality as an indicator of hospital quality: The problem with small sample size. JAMA. 2004;292:847–851. doi: 10.1001/jama.292.7.847. [DOI] [PubMed] [Google Scholar]
14.Russell MC, You YN, Hu CY, et al. A novel risk-adjusted nomogram for rectal cancer surgery outcomes. JAMA Surg. 2013;148:769–777. doi: 10.1001/jamasurg.2013.2136. [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Bilimoria KY, Stewart AK, Winchester DP, et al. The National Cancer Data Base: A powerful initiative to improve cancer care in the United States. Ann Surg Oncol. 2008;15:683–690. doi: 10.1245/s10434-007-9747-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Steele GD, Jr, Winchester DP, Menck HR. The National Cancer Data Base: A mechanism for assessment of patient care. Cancer. 1994;73:499–504. doi: 10.1002/1097-0142(19940115)73:2<499::aid-cncr2820730241>3.0.co;2-t. [DOI] [PubMed] [Google Scholar]
17.Winchester DP, Stewart AK, Phillips JL, et al. The national cancer data base: Past, present, and future. Ann Surg Oncol. 2010;17:4–7. doi: 10.1245/s10434-009-0771-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45:613–619. doi: 10.1016/0895-4356(92)90133-8. [DOI] [PubMed] [Google Scholar]
19.Best WR, Cowper DC. The ratio of observed-to-expected mortality as a quality of care indicator in non-surgical VA patients. Med Care. 1994;32:390–400. doi: 10.1097/00005650-199404000-00007. [DOI] [PubMed] [Google Scholar]
20.Hall BL, Hamilton BH, Richards K, et al. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: An evaluation of all participating hospitals. Ann Surg. 2009;250:363–376. doi: 10.1097/SLA.0b013e3181b4148f. [DOI] [PubMed] [Google Scholar]
21.Hougaard P. Frailty models for survival data. Lifetime Data Anal. 1995;1:255–273. doi: 10.1007/BF00985760. [DOI] [PubMed] [Google Scholar]
22.Giorgi R, Belot A, Gaudart J, et al. The performance of multiple imputation for missing covariate data within the context of regression relative survival analysis. Stat Med. 2008;27:6310–6331. doi: 10.1002/sim.3476. [DOI] [PubMed] [Google Scholar]
23.Ambler G, Omar RZ, Royston P. A comparison of imputation techniques for handling missing predictor values in a risk model with a binary outcome. Stat Methods Med Res. 2007;16:277–298. doi: 10.1177/0962280206074466. [DOI] [PubMed] [Google Scholar]
24.Agency for Healthcare Research and Quality. Guide to inpatient quality indicators. http://www.qualityindicators.ahrq.gov/downloads/software/sas/v21r4/iqi_guide_rev4.pdf.
25.Nicholas LH, Osborne NH, Birkmeyer JD, et al. Hospital process compliance and surgical outcomes in Medicare beneficiaries. Arch Surg. 2010;145:999–1004. doi: 10.1001/archsurg.2010.191. [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Hawn MT, Vick CC, Richman J, et al. Surgical site infection prevention: Time to move beyond the surgical care improvement program. Ann Surg. 2011;254:494–499. doi: 10.1097/SLA.0b013e31822c6929. [DOI] [PubMed] [Google Scholar]
27.Stulberg JJ, Delaney CP, Neuhauser DV, et al. Adherence to surgical care improvement project measures and the association with postoperative infections. JAMA. 2010;303:2479–2485. doi: 10.1001/jama.2010.841. [DOI] [PubMed] [Google Scholar]
28.Nagtegaal ID, van de Velde CJ, Marijnen CA, et al. Low rectal cancer: A call for a change of approach in abdominoperineal resection. J Clin Oncol. 2005;23:9257–9264. doi: 10.1200/JCO.2005.02.9231. [DOI] [PubMed] [Google Scholar]
29.Kapiteijn E, Marijnen CA, Nagtegaal ID, et al. Preoperative radiotherapy combined with total mesorectal excision for resectable rectal cancer. N Engl J Med. 2001;345:638–646. doi: 10.1056/NEJMoa010580. [DOI] [PubMed] [Google Scholar]
30.Sauer R, Becker H, Hohenberger W, et al. Preoperative versus postoperative chemoradiotherapy for rectal cancer. N Engl J Med. 2004;351:1731–1740. doi: 10.1056/NEJMoa040694. [DOI] [PubMed] [Google Scholar]
31.Morrow M, Katz SJ. The challenge of developing quality measures for breast cancer surgery. JAMA. 2012;307:509–510. doi: 10.1001/jama.2012.74. [DOI] [PubMed] [Google Scholar]
32.Keroack MA, Youngberg BJ, Cerese JL, et al. Organizational factors associated with high performance in quality and safety in academic medical centers. Acad Med. 2007;82:1178–1186. doi: 10.1097/ACM.0b013e318159e1ff. [DOI] [PubMed] [Google Scholar]
33.Jha A, Epstein A. Hospital governance and the quality of care. Health Aff (Millwood) 2010;29:182–187. doi: 10.1377/hlthaff.2009.0297. [DOI] [PubMed] [Google Scholar]
34.Kwon S, Florence M, Grigas P, et al. Creating a learning healthcare system in surgery: Washington State's Surgical Care and Outcomes Assessment Program (SCOAP) at 5 years. Surgery. 2012;151:146–152. doi: 10.1016/j.surg.2011.08.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
35.Wibe A, Eriksen MT, Syse A, et al. Total mesorectal excision for rectal cancer–what can be achieved by a national audit? Colorectal Dis. 2003;5:471–477. doi: 10.1046/j.1463-1318.2003.00506.x. [DOI] [PubMed] [Google Scholar]
36.American College of Surgeons Cancer Programs. Rapid Quality Reporting System (RQRS) http://www.facs.org/cancer/ncdb/rqrs.html.
37.American College of Surgeons Cancer Programs. Cancer Program Practice Profile Reports (CP3R) http://www.facs.org/cancer/ncdb/cp3r.html.
38.Lerro CC, Robbins AS, Phillips JL, et al. Comparison of cases captured in the national cancer data base with those in population-based central cancer registries. Ann Surg Oncol. 2013;20:1759–1765. doi: 10.1245/s10434-013-2901-1. [DOI] [PubMed] [Google Scholar]
39.Bilimoria KY, Bentrem DJ, Stewart AK, et al. Comparison of Commission on Cancer-approved and -nonapproved hospitals in the United States: Implications for studies that use the National Cancer Data Base. J Clin Oncol. 2009;27:4177–4181. doi: 10.1200/JCO.2008.21.7018. [DOI] [PubMed] [Google Scholar]

[B1] 1.Siegel R, Naishadham D, Jemal A. Cancer statistics, 2012. CA Cancer J Clin. 2012;62:10–29. doi: 10.3322/caac.20138. [DOI] [PubMed] [Google Scholar]

[B2] 2.Wibe A, Rendedal PR, Svensson E, et al. Prognostic significance of the circumferential resection margin following total mesorectal excision for rectal cancer. Br J Surg. 2002;89:327–334. doi: 10.1046/j.0007-1323.2001.02024.x. [DOI] [PubMed] [Google Scholar]

[B3] 3.Adam IJ, Mohamdee MO, Martin IG, et al. Role of circumferential margin involvement in the local recurrence of rectal cancer. Lancet. 1994;344:707–711. doi: 10.1016/s0140-6736(94)92206-3. [DOI] [PubMed] [Google Scholar]

[B4] 4.Marijnen CA, Nagtegaal ID, Kapiteijn E, et al. Radiotherapy does not compensate for positive resection margins in rectal cancer patients: Report of a multicenter randomized trial. Int J Radiat Oncol Biol Phys. 2003;55:1311–1320. doi: 10.1016/s0360-3016(02)04291-8. [DOI] [PubMed] [Google Scholar]

[B5] 5.Quirke P, Steele R, Monson J, et al. Effect of the plane of surgery achieved on local recurrence in patients with operable rectal cancer: A prospective study using data from the MRC CR07 and NCIC-CTG CO16 randomised clinical trial. Lancet. 2009;373:821–828. doi: 10.1016/S0140-6736(09)60485-2. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B6] 6.Fleshman JW. Multidisciplinary treatment of rectal cancer: The way of the future. JAMA Surg. 2013;148:778. doi: 10.1001/jamasurg.2013.2151. [DOI] [PubMed] [Google Scholar]

[B7] 7.Birkmeyer JD, Dimick JB, Birkmeyer NJ. Measuring the quality of surgical care: Structure, process, or outcomes? J Am Coll Surg. 2004;198:626–632. doi: 10.1016/j.jamcollsurg.2003.11.017. [DOI] [PubMed] [Google Scholar]

[B8] 8.Donabedian A. The quality of care: How can it be assessed? JAMA. 1988;260:1743–1748. doi: 10.1001/jama.260.12.1743. [DOI] [PubMed] [Google Scholar]

[B9] 9.Christian CK, Gustafson ML, Betensky RA, et al. The volume-outcome relationship: Don't believe everything you see. World J Surg. 2005;29:1241–1244. doi: 10.1007/s00268-005-7993-8. [DOI] [PubMed] [Google Scholar]

[B10] 10.Daley J. Invited commentary: Quality of care and the volume-outcome relationship—What's next for surgery? Surgery. 2002;131:16–18. doi: 10.1067/msy.2002.120237. [DOI] [PubMed] [Google Scholar]

[B11] 11.Sheikh K. Reliability of provider volume and outcome associations for healthcare policy. Med Care. 2003;41:1111–1117. doi: 10.1097/01.MLR.0000088085.61714.AE. [DOI] [PubMed] [Google Scholar]

[B12] 12.Birkmeyer NJ, Birkmeyer JD. Strategies for improving surgical quality: Should payers reward excellence or effort? N Engl J Med. 2006;354:864–870. doi: 10.1056/NEJMsb053364. [DOI] [PubMed] [Google Scholar]

[B13] 13.Dimick JB, Welch HG, Birkmeyer JD. Surgical mortality as an indicator of hospital quality: The problem with small sample size. JAMA. 2004;292:847–851. doi: 10.1001/jama.292.7.847. [DOI] [PubMed] [Google Scholar]

[B14] 14.Russell MC, You YN, Hu CY, et al. A novel risk-adjusted nomogram for rectal cancer surgery outcomes. JAMA Surg. 2013;148:769–777. doi: 10.1001/jamasurg.2013.2136. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B15] 15.Bilimoria KY, Stewart AK, Winchester DP, et al. The National Cancer Data Base: A powerful initiative to improve cancer care in the United States. Ann Surg Oncol. 2008;15:683–690. doi: 10.1245/s10434-007-9747-3. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B16] 16.Steele GD, Jr, Winchester DP, Menck HR. The National Cancer Data Base: A mechanism for assessment of patient care. Cancer. 1994;73:499–504. doi: 10.1002/1097-0142(19940115)73:2<499::aid-cncr2820730241>3.0.co;2-t. [DOI] [PubMed] [Google Scholar]

[B17] 17.Winchester DP, Stewart AK, Phillips JL, et al. The national cancer data base: Past, present, and future. Ann Surg Oncol. 2010;17:4–7. doi: 10.1245/s10434-009-0771-3. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B18] 18.Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45:613–619. doi: 10.1016/0895-4356(92)90133-8. [DOI] [PubMed] [Google Scholar]

[B19] 19.Best WR, Cowper DC. The ratio of observed-to-expected mortality as a quality of care indicator in non-surgical VA patients. Med Care. 1994;32:390–400. doi: 10.1097/00005650-199404000-00007. [DOI] [PubMed] [Google Scholar]

[B20] 20.Hall BL, Hamilton BH, Richards K, et al. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: An evaluation of all participating hospitals. Ann Surg. 2009;250:363–376. doi: 10.1097/SLA.0b013e3181b4148f. [DOI] [PubMed] [Google Scholar]

[B21] 21.Hougaard P. Frailty models for survival data. Lifetime Data Anal. 1995;1:255–273. doi: 10.1007/BF00985760. [DOI] [PubMed] [Google Scholar]

[B22] 22.Giorgi R, Belot A, Gaudart J, et al. The performance of multiple imputation for missing covariate data within the context of regression relative survival analysis. Stat Med. 2008;27:6310–6331. doi: 10.1002/sim.3476. [DOI] [PubMed] [Google Scholar]

[B23] 23.Ambler G, Omar RZ, Royston P. A comparison of imputation techniques for handling missing predictor values in a risk model with a binary outcome. Stat Methods Med Res. 2007;16:277–298. doi: 10.1177/0962280206074466. [DOI] [PubMed] [Google Scholar]

[B24] 24.Agency for Healthcare Research and Quality. Guide to inpatient quality indicators. http://www.qualityindicators.ahrq.gov/downloads/software/sas/v21r4/iqi_guide_rev4.pdf.

[B25] 25.Nicholas LH, Osborne NH, Birkmeyer JD, et al. Hospital process compliance and surgical outcomes in Medicare beneficiaries. Arch Surg. 2010;145:999–1004. doi: 10.1001/archsurg.2010.191. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B26] 26.Hawn MT, Vick CC, Richman J, et al. Surgical site infection prevention: Time to move beyond the surgical care improvement program. Ann Surg. 2011;254:494–499. doi: 10.1097/SLA.0b013e31822c6929. [DOI] [PubMed] [Google Scholar]

[B27] 27.Stulberg JJ, Delaney CP, Neuhauser DV, et al. Adherence to surgical care improvement project measures and the association with postoperative infections. JAMA. 2010;303:2479–2485. doi: 10.1001/jama.2010.841. [DOI] [PubMed] [Google Scholar]

[B28] 28.Nagtegaal ID, van de Velde CJ, Marijnen CA, et al. Low rectal cancer: A call for a change of approach in abdominoperineal resection. J Clin Oncol. 2005;23:9257–9264. doi: 10.1200/JCO.2005.02.9231. [DOI] [PubMed] [Google Scholar]

[B29] 29.Kapiteijn E, Marijnen CA, Nagtegaal ID, et al. Preoperative radiotherapy combined with total mesorectal excision for resectable rectal cancer. N Engl J Med. 2001;345:638–646. doi: 10.1056/NEJMoa010580. [DOI] [PubMed] [Google Scholar]

[B30] 30.Sauer R, Becker H, Hohenberger W, et al. Preoperative versus postoperative chemoradiotherapy for rectal cancer. N Engl J Med. 2004;351:1731–1740. doi: 10.1056/NEJMoa040694. [DOI] [PubMed] [Google Scholar]

[B31] 31.Morrow M, Katz SJ. The challenge of developing quality measures for breast cancer surgery. JAMA. 2012;307:509–510. doi: 10.1001/jama.2012.74. [DOI] [PubMed] [Google Scholar]

[B32] 32.Keroack MA, Youngberg BJ, Cerese JL, et al. Organizational factors associated with high performance in quality and safety in academic medical centers. Acad Med. 2007;82:1178–1186. doi: 10.1097/ACM.0b013e318159e1ff. [DOI] [PubMed] [Google Scholar]

[B33] 33.Jha A, Epstein A. Hospital governance and the quality of care. Health Aff (Millwood) 2010;29:182–187. doi: 10.1377/hlthaff.2009.0297. [DOI] [PubMed] [Google Scholar]

[B34] 34.Kwon S, Florence M, Grigas P, et al. Creating a learning healthcare system in surgery: Washington State's Surgical Care and Outcomes Assessment Program (SCOAP) at 5 years. Surgery. 2012;151:146–152. doi: 10.1016/j.surg.2011.08.015. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B35] 35.Wibe A, Eriksen MT, Syse A, et al. Total mesorectal excision for rectal cancer–what can be achieved by a national audit? Colorectal Dis. 2003;5:471–477. doi: 10.1046/j.1463-1318.2003.00506.x. [DOI] [PubMed] [Google Scholar]

[B36] 36.American College of Surgeons Cancer Programs. Rapid Quality Reporting System (RQRS) http://www.facs.org/cancer/ncdb/rqrs.html.

[B37] 37.American College of Surgeons Cancer Programs. Cancer Program Practice Profile Reports (CP3R) http://www.facs.org/cancer/ncdb/cp3r.html.

[B38] 38.Lerro CC, Robbins AS, Phillips JL, et al. Comparison of cases captured in the national cancer data base with those in population-based central cancer registries. Ann Surg Oncol. 2013;20:1759–1765. doi: 10.1245/s10434-013-2901-1. [DOI] [PubMed] [Google Scholar]

[B39] 39.Bilimoria KY, Bentrem DJ, Stewart AK, et al. Comparison of Commission on Cancer-approved and -nonapproved hospitals in the United States: Implications for studies that use the National Cancer Data Base. J Clin Oncol. 2009;27:4177–4181. doi: 10.1200/JCO.2008.21.7018. [DOI] [PubMed] [Google Scholar]

PERMALINK

Risk-Adjusted Pathologic Margin Positivity Rate As a Quality Indicator in Rectal Cancer Surgery

Nader N Massarweh

Chung-Yuan Hu

Y Nancy You

Brian K Bednarski

Miguel A Rodriguez-Bigas

John M Skibber

Scott B Cantor

Janice N Cormier

Barry W Feig

George J Chang