Comparison of publicly reported surgical outcomes with quality measures from a statewide improvement collaborative

To the Editor

The recent release of a “Surgeon Scorecard” has accelerated debate around the merits of publicly reporting surgical outcomes.¹ Based on Medicare claims from 2009 through 2013, this Scorecard provides the public with surgeon-specific complication rates for 8 elective procedures performed by nearly 17,000 surgeons. While the intent of this effort—greater transparency leading to better outcomes—is laudable, many contend that the Scorecard is misleading because it provides data for a single outcome measure that may not correlate well with other quality metrics.

We used data from the statewide clinical registry maintained by the Michigan Urological Surgery Improvement Collaborative (MUSIC) to evaluate this concern for one of the Scorecard procedures – radical prostatectomy (RP) for prostate cancer. We specifically examined whether surgeon-specific complication rates reported in the Scorecard correlate with several peri-operative quality measures endorsed by MUSIC urologists and patient advocates.²

Methods

Established with support from Blue Cross Blue Shield of Michigan, MUSIC is a consortium of 42 urology practices in Michigan that comprises nearly 85% of urologists in the state. The collaborative maintains a prospective clinical registry that includes detailed and validated intra-operative and post-operative clinical data obtained via medical record review by trained abstractors for all patients undergoing RP in participating practices.³

For this analysis, we identified every urologist in Michigan with both a risk-adjusted complication rate (i.e., in-hospital mortality or 30-day related readmission) for RP in the “Surgeon Scorecard” released by ProPublica, and outcomes data for 10 or more RPs in the MUSIC registry.

We then fit multivariable models to estimate risk-standardized, surgeon-specific performance on several metrics tracked in MUSIC to assess technical quality and recovery after RP.² These include blood loss, surgical margin status, pelvic complications, and 30-day readmissions and mortality. Using linear regression, we then examined the correlation between each MUSIC metric and the Scorecard outcome. Statistical testing was performed at the 5% significance level using electronic software (StataCorp, College Station, TX). Each practice obtained institutional review board approval of not-regulated status for collaborative participation.

Results

Among the 48 surgeons from Michigan with complication rates reported in the Scorecard, 40 participate in MUSIC, and 35 had data in the registry for at least 10 prostatectomies with more than 30 days of follow-up (Appendix). For this group (n=35), case volumes in the Scorecard and MUSIC registry ranged from 20–190 and 15–334, respectively.

Across four separate surgeon-specific outcomes, including a combined metric of 30-day readmissions and mortality analogous to that from the publicly-reported data, we noted only weak, non-significant correlations (R-squared ranges from 0.004 to 0.04) between measures from the MUSIC registry and complication rates from the Scorecard (Figure 1a–d).

Figure 1. Comparison of surgeon-specific peri-operative outcomes from the Surgeon Scorecard^a (x-axis) and MUSIC collaborative^b (y-axis).

a) 30 day readmission or mortality; b) Estimated blood loss; c) Surgical margin status for patients with organ confined disease (pathologic stage = T2)^c; and d) Pelvic complications^d

^aThe complication rate for the Surgeon Scorecard was a composite measure of in-hospital mortality or related hospital readmission within 30 days of surgery, adjusting for patient age, comorbidity, and surgical approach (i.e. open vs robotic)

^bFor MUSIC outcomes, the models are also adjusted for patient age, comorbidity, and surgical approach (i.e. open vs robotic), and account for clustering of outcomes within surgeons.

^cMargin data is reported for 34 of 35 MUSIC surgeons

^dPelvic complications are defined in MUSIC as the occurrence of intra-operative rectal injury or the requirement for urinary catheter or pelvic drain for greater than 16 or 2 days, respectively. Data for this measure has been collected since 4/1/14.

Discussion

For urologists in Michigan, we found no significant correlation between complication rates reported in the “Surgeon Scorecard” and peri-operative quality measures from a statewide improvement collaborative. This finding supports a prevalent concern that the limited data available in the Scorecard provides an incomplete, if not inaccurate, assessment of surgeon performance.⁴

There are, however, other potential explanations for our findings. First, because dates for included surgeries were not entirely congruent between the data sources, the absence of correlations could reflect changes in surgical performance over time. Second, we only examined data for urologists in Michigan; stronger correlations may exist for surgeons from other regions or for other procedures.

Nonetheless, our findings highlight limitations with emerging public reporting initiatives. Quality improvement collaboratives represent an established alternative that provide surgeons with comparative performance feedback on meaningful clinical and patient-reported outcomes, while linking these measures with specific improvement initiatives.⁵ Urologists in Michigan have used this model to achieve population-level improvements in several aspects of prostate cancer care,³ while avoiding potential inadvertent consequences of public reporting.⁶

Appendix. Surgeon case volumes in the MUSIC registry

MUSIC surgeon	No. of Prostatectomies	Months since collaborative entrya	Annualized prostatectomy rateb,c
1	24	36	7.9
2	23	32	8.5
3	15	16	11.3
4	26	26	12.1
5	30	29	12.4
6	24	21	14.0
7	38	31	14.7
8	42	34	14.9
9	45	36	15.0
10	49	36	16.4
11	53	37	17.4
12	55	36	18.5
13	44	29	18.5
14	63	36	21.1
15	70	36	23.4
16	56	28	23.9
17	51	25	24.6
18	53	24	26.9
19	69	11	29.8
20	27	28	30.3
21	64	24	31.4
22	38	14	31.5
23	66	25	31.6
24	35	12	33.7
25	105	36	34.6
26	56	19	35.6
27	81	23	41.8
28	74	21	41.9
29	83	22	45.4
30	46	12	47.1
31	196	37	63.5
32	137	25	65.0
33	197	36	66.3
34	323	37	104.7
35	334	26	155.2

^aSurgeon enrollment in in MUSIC occurred in a staggered fashion since 2011. This analysis includes all cases entered for each surgeon through 5/31/15.

^bEstimated average number of prostatectomies done per 12 months since collaborative entry

^cLinear regression analysis of annualized rates identified moderately strong correlation between case volumes from MUSIC and as reported in the Scorecard (R²=0.60, p<0.001)