The effect of consultant outcome publication on surgeon behaviour: a systematic review and narrative synthesis

Abstract

Introduction

Surgeon-specific outcome data, or consultant outcome publication, refers to public access to named surgeon procedural outcomes. Consultant outcome publication originates from cardiothoracic surgery, having been introduced to US and UK surgery in 1991 and 2005, respectively. It has been associated with an improvement in patient outcomes. However, there is concern that it may also have led to changes in surgeon behaviour. This review assesses the literature for evidence of risk-averse behaviour, upgrading of patient risk factors and cessation of low-volume or poorly performing surgeons.

Materials and methods

A systematic literature review of Embase and Medline databases was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. Original studies including data on consultant outcome publication and its potential effect on surgeon behaviour were included.

Results

Twenty-five studies were identified from the literature search. Studies suggesting the presence of risk-averse behaviour and upgrading of risk factors tended to be survey based, with studies contrary to these findings using recognised regional and national databases.

Discussion and conclusion

Our review includes instances of consultant outcome publication leading to risk-averse behaviour, upgrading of risk factors and cessation of low-volume or poorly performing surgeons. As UK data on consultant outcome publication matures, further research is essential to ensure that high-risk patients are not inappropriately turned down for surgery.

Introduction

Surgeon-specific outcome data or consultant outcome publication refers to public access to procedural outcomes for a named surgeon. It was first seen in cardiac surgery in New York State in 1991.¹ Over 20 years later, it was introduced to UK cardiothoracic surgery following the Kennedy inquiry into high mortality rates at Bristol Children’s hospital.² Consultant outcome publication was adopted by the remaining surgical specialties and interventional cardiology in 2012. The mandatory policy followed the Francis report into poor performance at Mid Staffordshire NHS Trust.³

The literature suggests that the introduction of consultant outcome publication has been associated with an improvement in patient outcomes in the United States and the UK.^4,5 However, it may have had other effects, both positive and negative, on the behaviour of surgeons. The aim of this review is to assess the ways that consultant outcome publication may have affected the behaviour of surgeons. With this in mind, this review assesses published literature for evidence of:

• whether consultant outcome publication has led to an increase in risk-averse behaviour among surgeons

• whether it has encouraged surgeons to register a higher number of risk factors for patients so that it appears that they are operating on higher-risk patients (i.e. ‘gaming’)

• whether consultant outcome publication has led to a reduction in the number of low-volume surgeons.

Methods

A systematic literature review of Embase and Medline databases was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. The following search terms were used: surgery OR surgeon AND outcomes AND report OR publication. All titles were viewed for relevant abstracts. Relevant abstracts were assessed to find all papers pertaining to potential change in surgeon behaviour associated with consultant outcome publication. All citations in chosen papers were searched for other articles that may include original data referring to the same. Only articles with original data were included. All titles returned were independently assessed for eligibility by two reviewers (MW/VM) for inclusion in the study. Any disagreements were settled by consensus. The primary assessments were evidence of risk-averse behaviour, evidence of gaming and cessation of operative activity by low workload volume or poorly performing surgeons.

Results

Our initial MeSH word search returned a total of 12,399 abstracts. Following review of these abstracts, 17 papers were included initially. Respective citation review yielded a further 8 studies, to give a total number of 25 studies included in our review (Fig 1; Table 1).

Figure 1.

Flowchart for study selection

Table 1.

Summary of studies including evidence of the effect of consultant outcome publication on surgeon behaviour.

Study	Location	Specialty	Survey (response)	Database	Included (n)	Outcomes/comments
Risk-averse behaviour
Evidence for
Schneider 19966	Pensylvannia	CTS/IC	Survey(64%/74%)		612/85	59% of cardiologists found that referring a high-risk patient for surgery had become more difficult since introduction of COP; 63% of cardiac surgeons were less willing to perform surgery on high-risk patients since introduction of COP.
Hannan 199724	New York	IC	Survey (36%)		450	14% of free text comments related to high-risk patients being turned down for surgery; 28% stated that report cards should be discontinued or not publically reported, with reasons given including that it deters surgeons from taking on high-risk patients
Burack 19997	New York	CTS	Survey (69%)		104	67% of respondents had refused surgery to at least one high-risk patient in the previous 12 months. 18% had refused surgery to 5 or more. Surgeons who had refused a high-risk patient due to concerns on the effect on their published outcomes were significantly more likely to have been practising as a consultant for less than 10 years, to have performed less than 100 cardiac procedures/year or to have less than 50% adult cardiothoracic practice.
Dranove 20028	New York and Pensylvannia	CTS		Medicare/AHD		Following COP, the health expenditure per patient in the year prior to intervention decreased suggesting that CABG was being performed on healthier patients.
Narins 200515	New York	IC	Survey		186	79% of cardiologists agreed or strongly agreed that COP had affected their decision to offer intervention to high-risk patients. Suggested not to include the highest-risk patients in the outcome data to help avoid risk averse behaviour.
Moscucci 20059	New York	IC		Regional PCI data	38,066	Patients from Michigan (non-reporting) were significantly more likely to undergo PCI for acute myocardial infarction and cardiogenic shock. Subsequently, patients from a public reporting state had a lower unadjusted in hospital mortality but this difference was not seen once risk adjustment for comorbidities was applied.
Apolito 200812	New York	CTS/IC		SHOCK registry	545	NY (public reporting) patients with AMI complicated with cardiac shock less likely to receive angiography, PCI and CABG compared with a non-reporting state. Lower prevalence of revascularisation in these patients was associated with a higher mortality (1.5-fold higher). NY patients had to wait 10 times longer to receive CABG (101.2 vs 10.3 hours).
Romano 201125	California	CTS		CCMRP/ CCORP	131,986	Following COP, high-mortality outlier hospitals were found to operate on patients with lower risk-adjusted mortality suggesting lower-risk patients were being selected.
Joynt 201210	New York, Pensylvannia, Massachussets	IC		Medicare		Patients with myocardial infarction were less likely to undergo PCI in states with public reporting with the greatest difference in intervention being seen in high risk patients (ST elevation myocardial infarction, cardiogenic shock or cardiac arrest). Following 5 years of COP, Massachusetts patients with cardiogenic shock had significant reduction in odds ratio for undergoing PCI compared with non-COP states.
Sherman 201326	Northwestern	Multi- specialty	Survey/ interview		122/20	78% were concerned that outcome publication of hospital data would lead to high-risk patients being turned down for surgery. 68% were concerned that outcome publication at hospital level may lead to the shifting of high-risk patients to safety-net hospitals.
Waldo 201513	New York/ Massachussets	CTS/IC		US NIS	84,121	Significantly fewer revascularisation procedures were performed in states with public reporting. Patients who underwent revascularisation procedures in public reporting states had a lower mortality compared with those in non-reporting states but were associated with a higher adjusted in-hospital mortality rate, with a higher proportion of mortality being associated with patients who did not undergo intervention.
Boyden 201511	New York and Michigan	IC		NCDR CathPCI registry	105,511	New York (public reporting) patients found to have a lower proportion of patients with high risk factors such as ST elevation myocardial infarction, non-ST elevation myocardial infarction, cardiac arrest and cardiogenic shock when compared to Michigan (non-public reporting). Significantly fewer New York patients had an extremely high risk (20%) of predicted risk of mortality, suggesting a degree of risk aversion.
Evidence against
Hannan 19944	New York	CTS		CSRS	57,187	COP associated with improved patient outcomes. Number of high-risk patients increased over first 4 years.
Hannan 199520	New York	CTS		CSRS	15,637	Actual mortality rate decreased during study period despite the fact that the average severity of illness of patients undergoing surgery increased (expected mortality 2.62–3.5%). 7.6% of patients with the highest risk were operated on by low-volume surgeons, which then dropped to 4.9% in 1991.
Peterson 199814	New York	CTS		Medicare	42,047	NY patients undergoing CABG were slightly older, more likely to have DM and to have had a preprocedural myocardial infarction when compared with non-PR states. From 1987 to 1992 there was an increase in percentage of patients 65–70 years receiving surgery from 3.4% to 8.4% which paralleled national trends.
Hannan 200323	US	CTS		Medicare	911,407	No evidence of risk averse behaviour or outmigration of patients. Some states with public reporting it was found to offer intervention for higher-risk patients.
Bridgewater 20075	UK	CTS		NWQI	25,730	Progressive increase in predicted mortality rate from 3.0 in 1997–98 to 3.5 in 2004–05 (P < 0.001). Median age of patients increased from 63 to 65 years (P < 0.001). Significant increases in the proportion of patients aged 80+ years, with renal dysfunction, recent myocardial infarction and peripheral vascular disease. Significantly more high-risk and very-high-risk patients underwent CABG after COP compared with before.
Li 201027	California	CTS		CCORP	36,923	Most hospitals had a higher or unchanged predicted risk of mortality suggesting a higher proportion of high-risk patients were being operated on. Proportion of high-risk patients undergoing CABG did not decline post COP.
Shahian 201519	US	CTS		STS	8,929	No change was seen across the preoperative risk factors) before and after the implementation of public reporting.
Gaming
Evidence for
Omoigui 199616	Cleveland	CTS		CCIR	42,027	Patients referred to Cleveland from New York were significantly more likely to have had previous open-heart surgery and to be of NYHA heart failure III or IV. There was a trend towards the transfer of a higher proportion of older patients from Cleveland to New York. Subsequently, there was a significantly higher postoperative morbidity and mortality in patients from New York. These findings were not seen between 1980 and 1988, the period prior to the introduction of COP.
Burack 19997	New York	CTS	Survey (69%)		104	30% of respondents admitted to a significant change in their practice, including altered profiling of patients (gaming), moving to thoracic non-cardiac surgery, relocation to a non-COP state or retirement from surgery. 40% of respondents stated that risk-factor gaming was the area requiring most improvement on the database.
Narins 200515	New York	IC	Survey (65%)		186	88% agreed or strongly agreed that physicians may report higher risk conditions for their patients features to improve risk-adjusted mortality outcome statistics.
Guru 200917	Ontario	CTS	Survey (95%)		52	84% of surgeons thought that upcoding of risk factors took place in Ontario following the introduction of COP.
Evidence against
Peterson 199814	New York	CTS		Medicare	42,047	No sign of outmigration over the time of the study with the percentage of 11.3% in 1992 being significantly less than the respective proportion from 1987 (P < 0.001). NY patients being referred out of state for CABG had similar demographics to those who remained in state. There was a significant decline for out of state referral from NY between 1987 and 1992. Elderly patients were less likely to be referred out of state for their procedure.
Hannan 200323	US	CTS		Medicare	911,407	No sign of outmigration.
Guru 200618	Ontario	CTS		CCN	67,693	No sharp increase in comorbidity following public reporting, suggesting no upcoding took.
Shahian 201519	US	CTS		STS	8,929	No change was seen across the preoperative risk factors before and after the implementation of public reporting. Across the 9 years, the expected mortality rate remained stable or increased slightly across both reporting and non-reporting regions.
Cessation of practice by low volume/poorly performing surgeons
Evidence for
Hannan 199520	New York	CTS		CSRS	15,637	Following COP, total number of surgeons performing less than 50 procedures per year fell from 39 to 33 (31% to 24%). The number of surgeons in the high-volume group increased from 30 to 41 (24–30%). Percentage of cases performed by low-volume surgeons decreased from 7.6% to 5.7% over the study period. Low-volume surgeons had a higher risk-adjusted mortality for each year studied.
Jha 200621	New York	CTS		CSRS		Poorer performing surgeons were more likely to leave CABG surgery in New York within 2 years of the publication of the report cards. More than 20% of those in the lowest quartiles stopped practising within this time frame. In contrast, only approx. 5% of the other quartiles left practice within the same time frame. 31 surgeons left practice between 1989 and 1999. Of the 25 who could be contacted, 18 responded. 10/18 stated COP had no effect on their decision to leave. 4 of the 10 had been in the lowest quartile prior to leaving. 2/18 said COP had a minimal impact. 6/18 said it had a moderate or substantial impact on decision to leave. Of these 8, 4 were in the lowest quartile.
Evidence against
Chen 201222	New York	IC		NY PCI	351	Cardiologists with poor performance were no more likely to leave practice than those with good performance.

AHD, American Hospital Association database; AMI, acute myocardial infarction; CABG, coronary artery bypass graft; CCIR, Cleveland Cardiovascular Information Registry; CCMRP, California CABG Mortality Reporting Programme; CCN, Cardiac Care Network Database; CCORP, California CABG Outcome Reporting Programme; COP, consultant outcome publication ; CSRS, Cardiac Surgery Reporting System; CTS, cardiothoracic surgery; IC, interventional cardiology; NCDR, National Cardiovascular Data Registry CathPCI Registry; NWQI, North West Quality Improvement Programme in Cardiac Interventions; NY PCI, New York PCI Reports; PCI, percutaneous coronary intervention; SHOCK, Should we emergently revascularise occluded coronaries for cardiogenic shock; STS, Society of Thoracic Surgeons Clinical Registry; US NIC, US National Inpatient Sample Database.

Risk-averse behaviour

Findings from survey-based studies strongly suggest that the introduction of outcome publication was associated with risk-averse behaviour among cardiac surgeons and physicians alike.^6,7 Dranove et al. used the intuitive step of reviewing inpatient expenditure in the 12 months prior to consideration of cardiac intervention as a proxy marker for patient’s illness severity, assuming high-risk patients would have incurred a greater cost in the year prior to their operation.⁸ Following consultant outcome publication, health expenditure decreased significantly more in public reporting states. Moscucci et al. found that patients from reporting states were significantly less likely to undergo percutaneous coronary intervention (PCI) for acute myocardial infarction (AMI) if there was associated cardiogenic shock.⁹ The patients from New York (reporting state) subsequently had lower unadjusted in-hospital mortality rate. However, this difference was not seen once comorbidities had been accounted for, suggesting that the difference was due to a higher proportion of high-risk patients being selected for PCI in non-reporting states.

A review of the Medicare database found that patients over 65 years admitted with myocardial infarction were less likely to undergo PCI in states with public reporting.¹⁰ The greatest difference in intervention was seen in those with high-risk patients such as ST elevation myocardial infarction and cardiogenic shock. Patients undergoing PCI in New York (public reporting) were found to have a lower proportion of high-risk factors for mortality when compared with Michigan (non-public reporting).¹¹ Subsequently, significantly fewer New York patients were found to have an extremely high risk of predicted risk of mortality (20%). Apolito et al. found that patients from states with outcome reporting who were admitted with acute myocardial infarction complicated by cardiac shock were less likely to receive angiography, PCI and coronary artery bypass graft (CABG) surgery.¹² A similar paper by Waldo used the US National Inpatient Sample Database and found that significantly fewer revascularisation procedures were performed in public reporting states.¹³

Studies using recognised databases provide evidence against the association of risk-averse behaviour with consultant outcome publication. Both US and UK studies have shown that the introduction of consultant outcome publication was associated with a progressive increase in expected risk-adjusted mortality and age over the same period, suggesting that high-risk patients had been operated on.^4,5 Using the EuroSCORE method, Bridgewater et al. showed more high-risk and very high-risk patients had undergone bypass surgery since the inception of outcome reporting.⁵

Using the Medicare database, Peterson et al.’s review of outcomes of patients over 65 years undergoing cardiac bypass surgery in New York State showed that the patients from New York were more likely to be older, have diabetes and to have had a preprocedure myocardial infarction when compared with those in states that did not operate public reporting.¹⁴

Gaming

Gaming has been defined as ‘the increased reporting of patients at high risk in an attempt to reduce risk-adjusted mortality’.¹⁵ Other forms of gaming include the referral of high-risk patients to non-reporting states (outmigration).¹⁶ Surveys of cardiac surgeons and interventional cardiologists revealed that a significant proportion of respondents were concerned that consultant outcome publication led to reporting of higher risk conditions for patients to improve risk-adjusted outcome statistics.^7,15,17 Following the introduction of consultant outcome publication to New York, there was a significant increase in the prevalence of renal failure, heart disease and chronic obstructive pulmonary disease among patients undergoing CABG surgery.¹⁸ The rate of chronic obstructive pulmonary disease (COPD) at one hospital increased from 1.8% to 52.9%. In contrast, state- and nationwide cardiac databases found that following the introduction of consultant outcome publication, there was no sharp increase in risk factors undergoing cardiac surgery.^19,20

Omoigui et al. suggested that consultant outcome publication had been associated with an increase in the number of high-risk patients transferred to neighbouring non-public reporting states (outmigration).¹⁶ Following the introduction of consultant outcome publication, patients referred to the Cleveland clinic from New York were significantly more likely to be older, have had previous open heart surgery and to be of NYHA heart failure III or IV since the introduction of consultant outcome publication. In contrast to Omouigui’s regional study, a retrospective review of all patients over 65 years undergoing isolated CABG surgery from the national Medicare database found no evidence of outmigration of patients from New York during the time of the study.¹⁴ In addition, patients who were referred out of state had similar demographics to those who received surgery in state.

Consultant outcome publication and cessation of low volume or poorly performing surgeon

A review of the first four years of the New York Cardiac Surgery Reporting System database showed that the proportion of surgeons performing less than 50 procedures/year fell from 31% to 24%.²¹ During the time of the study, 21 surgeons ceased practising CABG. Each surgeon had risk-adjusted rates of 2.5–5 times the state average in the year prior to cessation of practice. Jha et al. found more than 20% of surgeons in the lowest respective quartile for performance stopped practicing within two years of the introduction of consultant outcome publication.²² This is in contrast to only 5% of surgeons from the other quartiles leaving practice over the same time frame. Chen et al. found that cardiologists with poorer outcomes from PCI were no more likely to leave practice than those with good performance.²³

Discussion

Our review reveals conflicting evidence as to whether consultant outcome publication has led to risk-averse behaviour among surgeons. Several survey-based studies found that surgeons turned down high-risk patients for surgery due to concern regarding published outcomes. Such studies offer the benefit of including a large number of people. However, shortcomings include inaccurate recollection of respondents. This is evident in Guru et al.’s study,¹⁷ which showed 24% of responders believing that consultant outcome publication had led to risk-averse behaviour, but when the appropriate clinical dataset was reviewed there was no evidence of risk-averse behaviour.

Evidence contradicting the presence of risk-averse behaviour is based mainly on database studies. A common finding from these studies is that the risk factor profile of patients has increased since the introduction of outcome reporting and thus suggests there has been no risk-averse behaviour. However, a major flaw in these findings is that there is no control for the potential upgrading of risk factors that may have inadvertently obscured the presence of risk-averse behaviour. The collection of evidence is weakened further when considering the database studies included are from the cardiothoracic literature only and, on occasion, have used the same data on which to base their findings (Hannan et al.’s study²⁴ includes an extension of Peterson et al.’s work¹⁴ using the Medicare database). In addition, two of the studies are based on voluntary databases.^19,20 The voluntary, as opposed to mandatory, nature of these databases means that surgeons who submit data are more likely to be proponents of outcome publication and so may be less likely to show evidence of risk-averse behaviour.

Of the studies assessing risk-averse behaviour, only two have attempted to consider the outcomes for those patients who did not undergo intervention.^8,13 In doing so, the studies fail to address the bigger picture of not only whether risk-averse behaviour is present but also the effect that any risk-averse behaviour has on patient outcomes overall. Our review included three survey-based studies and one database study showing evidence of upgrading of risk factors with the evidence contrary to this coming from studies using databases.^7,15,17,18 Over 80% of respondents admitted to or had witnessed upgrading of risk factors.^15,17 With such a high proportion, it is concerning that this practice does exist and that the quantitative studies by Guru et al. and Shahian et al. were simply unable to pick up these practices within the methodology of their studies.^19,20

Omoigui et al.’s study suggesting outmigration been frequently quoted in the early literature on public reporting, owing to its relevance regarding potential gaming practices.¹⁶ However, this study was based on only 35 increased referrals/year from New York to Cleveland.²⁵ In addition, the first data from New York were from 1991 (previous data had been hospital-level data only) and were unexpected, so it is very unlikely that for the first two years of Omouigui et al.’s study (reviewed 1989–1993) referral practices would have been affected.²⁴

Our review found evidence of onsultant outcome publication leading to the cessation of practice in cardiothoracic surgery but not in the interventional cardiology literature.^21–23 The higher expected death rate associated with bypass surgery may have amplified the detrimental effects of public reporting on cardiac surgeons.

A weakness of our review is that much of the evidence is old, US-centric and, on occasion, has been based on the same databases. In addition, the databases used were either clinically or administratively maintained databases, which have specific strengths and weaknesses. Clinically maintained databases are more likely to contain the appropriate information as opposed to those maintained by non-clinical trained staff but this leaves such databases open to manipulation of risk factors by clinicians.

Implications for UK surgery

Burack et al.’s study is of note, as it showed that a higher proportion of junior surgeons had admitted to instances of risk-averse behaviour compared with their senior colleagues, highlighting the need for established, recognised mentoring roles for newly appointed surgeons performing complex surgeries.⁷

Within the finite resources of the NHS, the only way that such UK databases have any longevity is with surgeon-led data entry. In view of previous gross shortcomings in care, the drive for transparency should be encouraged. However, as long as public outcomes rely on clinician-led data entry, it will be very difficult to guarantee that upgrading of risk factors or selective reporting does not take place. Finally, it remains to be seen if lessons from these US studies can be extrapolated to UK surgery. As UK-based outcome data mature within the unique setting of surgeon-entered outcome data, there is a need for further research to assess whether risk-averse behaviour and gaming are issues. In doing so, we can help to ensure that the patients in most need of surgery, who stand to gain the most, do not miss out.