Publicly reported provider outcomes: The concerns of cardiac surgeons in a single-payer system

Abstract

BACKGROUND:

Provider outcomes reports are an important part of quality improvement efforts. The positive and negative impact of such reports on the delivery of care has not been extensively explored.

METHODS:

A survey of Ontario cardiac surgeons was performed in September 2003 to understand their concerns regarding performance reports. The questionnaire addressed the use of evidence-based practices, the impact of public-provider profiling on clinical practice and the improvement of current report cards. The survey was conducted with the distribution of a fiscal 2000/2001 cardiac surgery report card.

RESULTS:

There was a 95% (52 of 55 cardiac surgeons) survey response rate, of which 80% were high-volume surgeons with a case volume of more than 200 cases per year. Seventy-four per cent of surgeons had more than five years of experience. The majority of surgeons believed that performance reports influenced cardiologist referrals (84%) and patient choices (80%). A minority (48%) of surgeons believed that the reporting of in-hospital mortality was very or extremely useful, but a majority (83%) believed mortality rates indicated the relative performance of a cardiac surgeon. The majority of surgeons believed that routine upcoding of data (84%) and inadequate risk adjustment (75%) were weaknesses of present performance reports. Surgeons were divided regarding whether the institutional performance should continue to be publicly reported (51% agreed with public reporting).

CONCLUSIONS:

In a single-payer system, performance reports breed provider concerns similar to those seen in market-driven systems including high-risk patient avoidance and upcoding of data. Regardless, providers recognize that institutional performance reports, irrespective of public or confidential reporting, are important in continuous quality improvement.

The importance of public institutional performance reporting is widely recognized for the improvement of quality of care in all areas of medicine and surgery. In privately funded health care systems, good results in such reports may serve as a method of advertisement and may affect market forces for coronary artery bypass graft (CABG) surgery (1). However, this has not appeared to be the case in the United States, as surveys of cardiologists and cardiac surgeons indicate that very little attention is given to these reports in actual practice (2,3). If providers themselves do not support or trust report cards, neither public nor private profiling of performance is likely to enhance quality of care (4).

In Ontario, which has a single-payer system, cardiac surgery services are not thought to be market driven. Performance reports are primarily geared toward assisting self-motivated quality improvement. Therefore, it is extremely important that Ontarian providers believe in the credibility of the performance reports and support their public release.

Public reporting of risk-adjusted in-hospital mortality has been used in ranking the performance of institutions in the United States, Canada and the United Kingdom (5–7). There is some suggestion that CABG surgery performance reports have influenced the declines in mortality rates observed in regions of Canada and the United States (6,8–10). In the United States, however, market forces may also have encouraged the reactionary practices of high-risk case avoidance, and ‘gaming’ or upcoding of patient profiles (11,12). There are conflicting data regarding whether this phenomenon exists for specific high-risk patient subgroups (10,13,14).

Since 1999, quality assurance measures in Ontario have included the public reporting of in-hospital, all-cause mortality rates by institution for isolated CABG surgery. The present study evaluates the concerns of cardiac surgeons in our region regarding public reporting of cardiac surgery outcomes.

METHODS

Report card survey

Ontario has provided institutional cardiac surgery report cards since 1993 and publicly available reports since 1999. These institutional reports include crude and risk-adjusted mortality and length of stay data. Confidential surgeon-specific mortality rates have been provided to surgical division heads since 1994. In September 2003, all cardiac surgeons actively practising during the fiscal years 2000/2001 (April 1, 2000, to March 31, 2002) were confidentially surveyed. The survey coincided with the release of the 2003 cardiac surgery report card. The surgeon list was obtained by contacting each cardiac surgery department in Ontario. A written questionnaire with five-point Likert scales and open-ended questions was developed to collect data on the following major themes: background experience, present practices, departmental outcomes monitoring and thoughts on the current Ontario cardiac surgery report card. E-mail and telephone reminders were periodically sent to nonresponders until the second week of December. All survey response data were analyzed anonymously. A 95% (52 of 55 cardiac surgeons) response rate was achieved. For reference purposes, the present survey results have been compared with those of a 1995 survey (2) of a random sample of 36 cardiac surgeons (74% response rate) in Pennsylvania, USA, in which similar questions were asked three years after the initiation of public reporting.

Outcome data

Outcome data were derived for a retrospective cohort (n=67,693) of all patients who underwent isolated CABG surgery (without concomitant valve surgery) at the nine cardiac surgery institutions in Ontario between September 1, 1991, and March 31, 2002, from the Cardiac Care Network (CCN) database. A biannual chart audit (15) of the CCN database confirmed data accuracy to consistently be above 95%. The core covariates recognized in the literature (16) for risk adjustment of mortality are available from this database. As reported previously, using unique encrypted identifiers, patient-level data were linked to two other administrative databases (the Canadian Institute for Health Information and the Registered Persons Database) for further clinical information such as in-hospital and out-of-hospital death up to 30 days after surgery.

Analysis

Logistic regression models were built to determine the risk-adjusted 30-day mortality rates by fiscal year. The expected death rate was calculated by 30-day mortality logistic regression models. Variables for risk adjustment included previously identified core variables (eg, age, sex, coronary disease pattern, Canadian Cardiovascular Society angina class, diabetes, left ventricular function, reoperative CABG surgery, triage status, socioeconomic status and the Charlson comorbidity index) (16). Covariates, including diabetes, chronic obstructive pulmonary disease, cerebrovascular disease, peripheral vascular disease and dialysis, were initially derived by linkage to hospital discharge abstracts for fiscal years 1991 to 1995 and directly from the CCN database for fiscal years 1996 to 2001.

The survey results were linked anonymously through CCN physician numbers to the patient database to understand the relationship between surgeons’ survey responses and surgeon-specific case-mix (the distribution of comorbidities in the patient population of each surgeon). High-risk patients were defined as those in the upper tercile of predicted 30-day mortality risk based on the 1995 to 2001 period. Surgeon-specific three-year adjusted mortality rates were also analyzed. A programmer independent of the survey project completed this linkage to ensure that all surgeon-specific survey and outcome results remained anonymous.

RESULTS

Report card survey

The majority of Ontario cardiac surgeons had been in practice for more than five years (74%), had total case volumes greater than 200 cases per year (80%) and had isolated CABG surgery volumes greater than 100 cases per year (79%). Surgeons reported the use of evidence-based practices including the use of the internal mammary artery graft in more than 90% of patients (94%), the prescription of acetylsalicylic acid on discharge (94%), the prescription of a beta-blocker on discharge (83%) and the prescription of a lipid-lowering medication on discharge (81%). Table 1 summarizes the survey results of Ontario cardiac surgeons compared with a similar survey (2) of cardiovascular specialists in the state of Pennsylvania. Ontario surgeons were divided (49% opposed) on the public reporting of hospital-specific outcomes and strongly opposed (74%) to public reporting of surgeon-specific outcomes (Table 1). The majority (62%) of surgeons who had surgeon-specific mortality rates that were above the 50th percentile in Ontario supported public reporting of hospital-specific rates, whereas a minority (36%) of surgeons who had mortality rates that were lower than the 50th percentile in Ontario supported public reporting. The majority of Ontario surgeons agreed that mortality rates were useful in monitoring quality of care (73%), and important in assessing the relative performance of a surgeon (83%). In contrast, only a minority (32%) of surgeons in Pennsylvania, where public reporting of surgeon-specific results is mandated by the government, believed that mortality rates were important in assessing relative surgeon performance (2).

TABLE 1.

Views of cardiac surgeons in Ontario versus Pennsylvania (United States) (2) regarding reporting of outcomes for coronary artery bypass graft surgery

Question (likert response scale value reported)	Ontario, % (n=52)	Pennsylvania, % (n=36)
Do you support the public release of hospital-specific outcomes? (Yes)	51	–
Do you support the public release of surgeon-specific outcomes? (Yes)	26	–
Do you find reporting of risk-adjusted in-hospital mortality rates useful in monitoring quality of care? (useful)	73	86†
How important are risk-adjusted mortality rates in assessing the relative surgeon performance? (important)	83	32†
Do you think that public reporting is important in influencing referral patterns of cardiologists? (important)	84	13*
Do you think that public reporting is important in influencing patients choosing a cardiac surgeon? (important)	80	–
Do you slot high-risk patients to those surgeons who have better results or are more senior? (often)	66	–
Has the reporting of outcomes changed your willingness to operate on high-risk patients? (less willing)	24	63†
What do you believe are the limitations in accurately assessing surgical performance in outcomes reports?
Insensitive outcomes	27	78†
Inadequate risk adjustment	75	85†
Unreliable data sources	71	57†
Do you think any hospitals routinely upcode patient disease status and comorbidities in the data collected? (Yes)	84	–
Do you think adequate data for risk-adjusting of surgical outcomes is collected? (Yes)	39	–
What responses have you made in your practice in response to the institutional report cards?
Improved record keeping	17	–
Standing orders/care maps	10	–
Created a database	8	–
Audited charts to ensure evidence-based practices	6	–
Revised standing orders	6	–

^*Survey of a random sample of cardiologists in the state of Pennsylvania;

^†Survey of a random sample of cardiothoracic surgeons in the state of Pennsylvania

Most surgeons believed that Ontario’s institutional public report cards influenced cardiologist referral (84%) and patient choice (80%). In contrast, in Pennsylvania, only a minority (13%) of cardiologists and cardiothoracic surgeons believed that cardiology referral patterns were influenced by public reporting (2). Many Ontario surgeons (66%) also reported that high-risk patients were slotted to senior or more experienced surgeons, while a minority (24%) admitted that some high-risk patients were avoided because of report cards. A higher percentage of Pennsylvania surgeons (63%) acknowledged high-risk case avoidance (Table 1). The proportion of high-risk patients remained similar during confidential reporting (1995 to 1998, 33%) and public reporting (1999 to 2001, 35%). A c statistic of 0.77 was achieved for the 30-day logistic mortality model from which the expected risk of patients was derived. There was no difference in the proportion of high-risk patients operated on by surgeons who reported high-risk case avoidance (36%) compared with those who did not (35%) during public reporting (P=0.65).

The majority of Ontario surgeons had at least one major criticism of the current CABG surgery report card including inadequate risk adjustment of outcomes and unreliable data sources (Table 1). Pennsylvania surgeons were concerned about inadequate risk adjustment on performance reports but also worried that the outcomes monitored were insensitive (Table 1). In Ontario, many surgeons suspected that some of the data were upcoded (84%). Upcoding refers to the purposeful inflation of patient disease severity coding and results in risk-adjusted outcome rates appearing more favourable. Many surgeons were concerned that inadequate data for risk adjustment was collected (61%). A minority of Ontario cardiac surgeons (17%) reported changing their clinical practice in response to institutional report cards, including improved recordkeeping, creating standing orders, creating a database, auditing charts to ensure evidence-based practices were employed and revising standing orders (Table 1).

The individual comments of surgeons regarding the concept of publicly reporting outcomes for CABG surgery are summarized in Table 2. These comments are summarized under major themes that arose, including benefits of public accountability, stigma of an outlier label, high-risk case avoidance, accuracy of the report card and misinterpretation by the public of public reports. Table 3 includes provider suggestions on ways in which the current CABG surgery performance report could be improved to be more clinically relevant to providers.

TABLE 2.

Comments from cardiac surgeons regarding public release of a coronary artery bypass graft surgery report card in Ontario

Positive comments
Public accountability and improvement in quality of care
One of the few accountability assessments in health care at present
Find nothing wrong with transparency
Public right to know and choose improved quality so surgeons cannot hide behind false impressions
I believe this may influence referral patterns in a positive way, will encourage self-assessment of surgeons practice
It can only improve care
Cardiac surgery is now a complacent specialty mired in the past. It needs a kick to get up to speed
They pay for the system equals their right
Negative comments
Stigma of an outlier label
Has potential detrimental effects that can be long-lasting even after issues have been corrected
Unfair propaganda. Good work is done by all institutions in Ontario (ie, good people are found everywhere)
Can be very misleading and would negatively influence practice
High-risk case avoidance
I will not do high risk-cases – who will?
You report results, I guarantee you high-risk patients will be treated with palliative medical therapy: A disaster!
It will lead to decreased patient access to surgery
Is it correct to have disincentives to perform surgery on patients with high-risk cardiac problems?
Are we denying people with a 70% to 90% chance of surviving because they have a 10% to 30% chance of death?
So what do we do, stop doing refusals if your statistics do not reflect this?
Accuracy of report card
Even though reports are ‘risk-adjusted’, I believe they may be misleading. Mortality is not the most important index
Not needed, the Canadian health care system has managed to do well without the option of choosing your health care provider at the specialty level
Certain institutions in Ontario have a long record of falsifying statistical data
Only statistics lie and this is grossly unfair to all health care providers
Misinterpretation by public, press and government
Insufficient education, understanding of statistics by press and lay public
Does the public understand the concept of ‘risk-adjusted’ versus crude rates?
General public do not understand reasons well enough to make them meaningful for them. It will confuse the average person…ie, Who is the better surgeon, surgeon with crude 0.4%, risk-adjusted 0.6% or one with crude 4%, risk-adjusted 2.5% mortality
Year-to-year variations and public interpretation are potentially misleading issues
It is more important for the surgeon and department to be aware and put in place ‘checks’ to improve quality. Lay people see only a percentage mortality and assume that is either good or bad without further understanding

TABLE 3.

Cardiac surgeon suggestions on improvements for the coronary artery bypass graft surgery report card in Ontario

You need to have independent data collectors who do not have any vested interests in a given institution...Strict guidelines for assessing hospital charts for chronic obstructive pulmonary disease, peripheral vascular disease, etc

More data is always better. Often, data collected preoperatively at referral need to be continuously updated throughout the stay to ensure that it is properly coded

Long-term outcomes are important

There is no score for diffuseness of [coronary] disease, this is very hard to objectively score, yet likely an important predictor of outcome

Despite benefits described in the literature about advances such as point-of-care blood testing or continuous dialysis, present monitoring has not given any ammunition to obtain these products

Add valve surgery and other complex procedures

DISCUSSION

Provider performance reports can have both positive and negative effects on the delivery of health care. We surveyed Ontario cardiac surgeons to determine their views on institution-level postoperative mortality report cards, and generally found a higher level of support for some aspects of public reporting than was reported previously in Pennsylvania (2).

The high degree of confidence that Ontario surgeons have in the current performance report cards may reflect various factors. First, the performance reports were developed collaboratively and voluntarily with Ontario surgeons by a health services research agency that is at arm’s length from the government. This contrasts with state-mandated and contracted report cards in American jurisdictions. Second, the Ontario reports are at the institutional level; presumably, surgeons are more threatened by the surgeon-specific reports in Pennsylvania and other American states. Third, the relatively low supply of cardiac surgery services in Ontario means that surgeons are less likely to see market share shifts in response to publicity about institutional outcomes. Fourth and finally, because cardiac surgery is already regionalized and limited to high-volume surgeons and centres in Ontario, the differences in outcomes across centres have been relatively modest. All these factors provide a very different provider perception of performance reports in Ontario than in American states where government-mandated reporting exists.

Nonetheless, even in Ontario, where fixed capacity and regionalization have blunted market forces, many surgeons believed that public reporting of institutional results influenced referral patterns and patient choice. Paradoxically, in Pennsylvania, which has a market-driven health care system, a minority of surgeons believed that reports influenced cardiologist referral patterns, perhaps because of research reports suggesting that the state-wide report cards drew limited readership and attention from cardiologists (2). In Ontario, referral patterns have predominantly been geographically based, with most patients referred to the centre closest to where they live. The exception is the Greater Toronto Area, where several centres are operational and there is an overlap in geographical referral regions. Wait times for cardiac surgery are generally short across Ontario and, therefore, have not been a significant factor in influencing referrals. Ontario public reports on cardiac surgery are not widely promoted and it is likely that many cardiologists and patients are relatively unaware of the results.

On the other hand, surgeons in Pennsylvania reported (2) a higher rate of high-risk case avoidance than those in Ontario, suggesting that public reporting of surgeon outcomes may impair patient access to services. A potential limitation of the comparison made in the present report is the fact that the Pennsylvania survey was conducted in an earlier era. However, it is interesting to note that in Ontario, no objective evidence of a decrease in the proportion of high-risk patients operated on during the era of public reporting was found, even among surgeons who reported doing so with some cases. Perhaps no measurable trend was evident because survey responses reflected decisions that had been made on a small number of individual patients, although this had been perceived by some surgeons as a significant change in their practice. It is also conceivable that surgeon survey perceptions of risk avoidance were based on unmeasured confounding variables that have not been identified as important for risk adjustment.

High-risk case avoidance was also evident from the results of a survey in New York, USA (17). In the survey, New York cardiac surgeons were more likely to turn down high-risk coronary surgeries than aortic dissection procedures of similar risk for which there was no public outcomes report. In New York state, it was found that high-risk case avoidance was more prevalent in less experienced surgeons (17,18). In spite of public reporting of mortality rates for coronary surgery in the state of New York, high-performing hospitals and surgeons have not increased their market share of referrals and, in fact, even a few high-performing surgeons admitted to leaving the state due to “lack of enjoyment” in devising plans to ensure their reported mortality rates remained low (19).

In New York, both the institutional and surgeon-specific risk-adjusted mortality rates significantly decreased over time, in part attributed to negative factors such as the retirement or emigration of outlier surgeons, changes in case-mix to lower-risk patients, and the upcoding of patient risk profiles (5,19). The upcoding of cardiac risk factors was observed by a drastic increase in the reporting of comorbidities from 1989 to 1990/1991, when results became publicly reported (11). Of 109 surgeons in New York (69% response rate), more than 37% of cardiac surgeons perceived that, as a result of public reporting, a significant proportion of their colleagues had changed the patient profile that they deemed operable, changed their practice from cardiac to predominantly thoracic surgery, and relocated to another state or retired. The survey also demonstrated that surgeons were not aware of comorbidity definitions, as demonstrated by a significant proportion choosing the wrong definition for chronic obstructive pulmonary disease (49%) (17). New York surgeons indicated that ‘gaming’ on the report card required improvement (40%) (17). This concern regarding ‘gaming’ in New York did not just exist in the collection of comorbidity data but also in the alteration of clinical practice including upgrading of procedures (eg, inserting a preoperative intra-aortic balloon pump or performing an additional procedure such as a ventricular aneurysm repair by applying a few sutures) and/or the expeditious transfer of moribund patients to other care facilities before death (17). In response to this, the New York state cardiac surgery advisory board began to audit procedures to help capture and correct such manipulations (17). In Ontario, we have observed some differences in comorbidity coding at the hospital level, which chart audit has demonstrated to reflect misinterpretation of comorbidity definitions.

Many of the concerns of providers in Ontario regarding the limitations of performance reports for CABG surgery were similar to those of surgeons in states with mandatory reporting. A significant proportion of Ontario providers believed that there was inadequate risk adjustment of outcomes in performance reports (2). In Ontario, risk adjustment for mortality has been extensively investigated, and it has been shown that only a few core covariates are necessary to discern institutional performance (16). When asked to suggest variables that could be added to the adjustment model, surgeons recognized that many fundamental clinical characteristics were not objectively measurable (eg, relative diffuseness of coronary artery disease, relative frailty of a patient). Furthermore, mortality was not believed to be totally reflective of quality of care in either Ontario or Pennsylvania (2). Many Ontario providers raised concerns about coding practices regarding variables used in risk-adjusting mortality, despite no indication of up-coding in past CCN data audits (15). This feedback on data indicates the need for independently collected data with strict definitions in the production of outcomes reports for any region embarking on a quality-monitoring initiative that is meant to be transparent to all stakeholders. Ontario has worked toward standardizing definitions through consensus panel input from across the province. However, despite these efforts, even in current data, there are some unexpected differences in rates of certain variables that may affect risk-adjusted mortality rates. Continuous independent auditing of comorbidity data is required to ensure that performance reports remain accurate in demonstrating the relative performance of providers.

The comments received from providers regarding outcomes report cards for CABG surgery in Ontario mainly focused on the negative implications of public reporting. It may be that confidential performance reporting is sufficient to shift provider focus to the goal of improving quality of care. To change practice culture, performance report cards must evolve to be credible and relevant to providers as tools for learning, rather than increasing paranoia and mistrust. The end goal for performance reports should be improved care.

In Ontario, cardiac surgeons, after much debate, collectively agreed in 1999 to publicly release institutional outcomes during a year when there were no outlier institutions. It is interesting that despite a consensual process of public release, surgeons appear to have developed the same concerns as regions in the United States that are forced by law to publicly report (5,20). Approximately four years after the initiation of public reporting in Ontario, it appears that surgeons are divided about the public release of institutional performance reports. Surgeons who believe in public release recognize that a public health care system should be accountable to the taxpayer and outcome reporting is part of that equation. It has been noted that patients are also interested in performance measures not listed by providers, such as narratives of past patient experiences with the provider, the average time spent with each patient and the quality of communication with patients (21,22). The present study focused on the provider perspective; however, if patients are also expected to use these reports, performance measures must reflect issues important in their decision-making process (21,22).

In Ontario, many cardiac surgeons are still worried about the stigma attached to being labelled as an outlier institution. It is evident from our feedback that the lasting effects of an ‘outlier’ label are concerning to surgeons and may continue to discourage providers who have successfully improved their performance over the long term. One-quarter of surgeons worried that with public reporting, high-risk case avoidance would impair patient access to surgery. However, we found no objective evidence of this in the case-mix of their patients during the era of public reporting. As our population of CABG surgery candidates becomes older and more complex, this issue will increasingly become important. Our survey provided a limited snapshot of how public reporting may impact on high-risk case avoidance. Furthermore, prospective research focusing on this issue is required to accurately measure its prevalence. We have previously noted (23) that the mode of public reporting of hospital-specific mortality rates in our region did not improve outcome results further than those achieved with confidential reporting. Perhaps it can be argued that public reporting, especially when singling out individual providers, may do more harm than good in terms of denying high-risk patients access to care wherein providers are careful to only accept those patients with lower risk profiles. In New York state, researchers have noted that patients with emergent cardiac ischemia (ie, acute myocardial infarction with shock) may have been denied invasive revascularization strategies since the inception of public reporting (24).

Therefore, it is not surprising that while cardiac surgeons in Ontario generally were in favour of the concept of report cards, many were opposed to the public disclosure of results, more so for the release of surgeon-specific data (74%) than institutional results (49%). The opposition to public reporting may be related, in part, to the fear of the unknown because, to date, only institutional results are available to the public, while surgeon-specific results continue to be confidential. There could be concern that the potential downsides of disclosure would be greater at the surgeon than the institution level, and that the results may be less valid due to the smaller surgical volumes and the inherent inaccuracies of the risk-adjustment models. Paradoxically, support for public reporting was greater among surgeons with high risk-adjusted mortality rates. Finally, there are likely commonalities in terms of patient care for all patients undergoing surgery within any individual hospital, although we acknowledge that surgical referral patterns may exist within institutions designed to optimize overall results (ie, the referral of high-risk patients to more experienced surgeons).

The goal of risk-adjusting mortality reports is to level the playing field for comparisons among surgeons or institutions by focusing on the common risk factors that separately and together increase the risk of postoperative death. Unfortunately, the efficiency of the statistical model for fair comparisons can be undercut by rare preoperative catastrophes or combinations of severe risk factors that are either not routinely considered or incompletely weighted. Because these factors are rare, and postoperative deaths are also rare, many years of data may be needed to create a high degree of statistical confidence in intersurgeon comparisons. As a consequence, outcome rates are likely more valid at the institutional level than at the level of the individual provider.

It seems only prudent to respond to the concerns of cardiac surgeons in improving the accuracy and comprehensiveness of future report cards, especially if they are to be publicly reported and impact change in the practices of our surgeons. In an era of public accountability, it would be difficult to revert to private reporting. We are striving toward this goal by working with our stakeholders to develop and collect consensus quality measures with a rigorous independent data audit (25). In addition, rather than simply measuring outcome rates, our region will have provided more detailed independent feedback on how adverse outcomes relate to quality of care problems (26). We anticipate that providing clinically relevant feedback will provide a clearer roadmap to direct quality improvement activities. In recent years, the mortality rates in our province, after levelling off at 2% for a number of years, have improved significantly (23). The current crude in-hospital CABG mortality rate since fiscal year 2002 is approximately 1% without public reporting of surgeon-specific results, and now we have changed our focus to other important quality indicator rates that we found to have significant variation, likely reflective of differences in quality (25).