Endorsing performance measures is a matter of trust

Abstract

Healthcare agencies should ensure that performance measures for accreditation or reimbursement are supported by sound scientific evidence and safeguarded from potential commercial influences, argue Peter Eichacker and colleagues

Key messages.

• The US National Quality Forum endorses performance measures that the Centers for Medicare and Medicaid Services can then adopt nationally

• The forum’s endorsement of a bundle of performance measures for managing sepsis, including the need for central venous pressure and blood oxygen saturation measures, failed to adhere to the agency’s own standards regarding the soundness of supporting scientific evidence and freedom from potential commercial influences

• Unless government can ensure that its endorsements adhere to these standards, trust in the performance measures it endorses will be undermined

• Commercial representatives should have no role in endorsing performance measures

Using evidence based performance measures to guide patient treatment is advocated to improve outcomes and decrease costs.¹ The US Centers for Medicare and Medicaid Services uses performance measures to regulate US healthcare by withholding reimbursement and accreditation from non-compliant providers.^{2 3 4} With such compelling incentives, the performance measures essentially become healthcare mandates.

The National Quality Forum evaluates and endorses performance measures for the Centers for Medicare and Medicaid Services, ensuring that they are supported by strong, unbiased, scientific evidence.⁵ The NHS’s Quality and Outcome Framework has a similar role.^{6 7}

SEP-1 is a performance measure for managing patients with sepsis that was endorsed by the forum and adopted by the Centers for Medicare and Medicaid Services while controversial interventions were still undergoing clinical testing. Even after this testing showed lack of efficacy, the interventions were not removed. Furthermore, a trade organisation with direct ties to a manufacturer standing to benefit from SEP-1 took part in the endorsement. Without rigorous, independent scientific review protected from potential industry influence, endorsements like that of SEP-1 will undermine trust in governmental agencies and compromise patient care.

National Quality Forum endorsement process

The forum is a non-profit, non-partisan agency founded in 1999, with member organisations including medical societies, healthcare corporations, insurers, trade associations, and manufacturers. It publicly requests the voluntary submission of performance measures for selected areas of healthcare and then determines through consensus whether they meet the agency’s endorsement criteria (box 1).^{8 9 10 11 12}

Box 1. Summary of the National Quality Forum’s endorsement process.

A steering committee of 20-25 people, who can be nominated by and from member organisations, is central to the endorsement process.⁸ Nominees cannot have any financial or other interest that could “actually, or be perceived to, significantly impede (the nominee’s) objectivity or create an unfair competitive advantage for (the nominee) or organization associated with (the nominee).”⁸ For approval there must be “unambiguous evidence” that the performance measure improves health.⁹ Randomised controlled trials providing precise, unbiased effect estimates are deemed high quality evidence. Those with bias flaws or non-randomised trials with imprecise effect estimates or that do not control for confounders are considered low quality evidence. For composite measures, the forum requires that each component intervention meets the criteria for benefit.¹⁰ Performance measures approved by the committee are submitted for public comment and then evaluated by the forum’s member organisations and the Consensus Standards Approval Committee. The forum’s board of directors decides on final endorsement.⁸ More than 300 performance measures have been endorsed by the National Quality Forum and adopted by the Centers for Medicare and Medicaid Services.^{11 12}

Bundle of performance measures for sepsis

In 2008 the forum endorsed a performance measure for managing sepsis submitted by Henry Ford Hospital (table 1).^{13 14} But it did not endorse two interventions in the measure—central venous pressure and blood oxygen saturation within six hours of a patient presenting with septic shock. In 2012 Henry Ford Hospital submitted an application for a new sepsis measure: SEP-1. Besides stipulating a larger fluid challenge and serial lactate measures, it differed little from the one originally submitted in 2008 and again required measurement of central venous pressure and blood oxygen saturation.¹⁵

Table 1.

Timeline of events related to SEP-1

Date	Henry Ford Hospital	NQF and CMS	Confirmatory trials of CVP and ScVO2 measures	Appeals from professional groups
2008	Original performance measure submitted	NQF infectious disease committee reviews original measure
Mar 2008			ProCESS starts
Oct 2008		NQF endorses original measure but not CVP and ScVO2 measures	ARISE starts
Feb 2011			ProMISe starts
Jun-Aug 2012	Original measure, now called SEP-1, submitted for re-endorsement with CVP and ScVO2 reinserted	NQF infectious disease committee begins SEP-1 review		American College of Emergency Physicians questions CVP requirement and requests that full endorsement awaits confirmatory trial results
Oct-Dec 2012		NQF member and public comment period regarding SEP-1		American College and Society of Academic Emergency Physicians and Cleveland Clinic doctors question SCVP and ScVO2 requirements and state that more studies are necessary before endorsement
Mar 2013		NQF endorses full SEP-1 including CVP and ScVO2 measures
Apr 2013				Seven societies* question CVP and ScVO2 requirements and request that the measures’ endorsement awaits confirmatory study results
May 2013			ProCESS completes enrolment
Jun 2013		NQF finalises SEP-1 endorsement and CMS requires participating hospitals to collecting data for SEP-1 in Oct 2015
Mar 2014			ProCESS reports that CVP and ScVO2 measures do not improve sepsis care
Apr 2014		NQF patient safety committee removes CVP and ScVO2 measure from SEP-1	ARISE completes enrolment
Jul 2014			ProMISe completes
Aug 2014		CMS delays SEP-1 implementation and NQF asks developers and stakeholders to revise SEP-1s requirement for CVP and ScVO2 measures
Sep 2014		NQF patient safety committee approves and NQF later endorses revised SEP-1, which includes CVP and ScVO2 as options among other untested measures
10/2014			ARISE reports that CVP and ScVO2 measures do not improve sepsis care
Mar 2015		CMS again requires participating hospitals to begin SEP-1 data collection in Oct 2015 including data regarding CVP and ScVO2	ProMISe reports that CVP and ScVO2 measures do not improve sepsis care

^*American College of Chest Physicians, American Association of Critical Care Medicine, Greater New York Hospital Association, Midwest Critical Care Collaborative, National Association Medical Directors of Respiratory Care, Society of Academic Emergency Medicine, and Society of Hospital Medicine. ARISE=Australian resuscitation in sepsis evaluation trial; CMS=Centers for Medicare and Medicaid Services; CVP=central venous pressure; NQF=National Quality Forum; ProCESS=protocolised care for early septic shock trial; ProMISe=protocolised management in sepsis trial; ScVO₂=blood oxygen saturation.

In contrast to 2008, in March 2013 the National Quality Forum endorsed all of SEP-1 including the requirement for central venous pressure and blood oxygen saturation (table 1).^{13 15} At the time, there was controversy regarding potential financial conflicts (see later) and the validity of the analysis of the seminal study supporting this requirement.^{16 17 18} A body of published literature not supporting a physiological basis for central venous pressure and blood oxygen saturation in septic shock management^{19 20 21 22} led many sepsis experts to question these measures’ usefulness.^{23 24 25 26 27} Two large studies had shown low physician compliance with the Surviving Sepsis Campaign guidelines, include these measures.^{24 28 29 30}

Three multicentre randomised trials were started in 2008 to investigate the usefulness of measuring central venous pressure and blood oxygen saturation for sepsis and were nearing completion at the time of SEP-1’s endorsement.³¹ During the endorsement process eight professional societies raised concerns and asked the forum to wait for the results of the ongoing trials before endorsing SEP-1.^{13 32 33 34} Based on the forum’s 2013 endorsement, the Centers for Medicare and Medicaid Services announced that it would adopt all of SEP-1 for its hospital inpatient quality reporting programme, thereby requiring many doctors to report on their use of SEP-1’s interventions, whether or not they judged them best for their patients.³⁵

Evidence provided in the SEP-1 application

The application submitted for SEP-1’s endorsement in 2012 included four sources of evidence to support the measurement of central venous pressure and blood oxygen saturation: the Surviving Sepsis Campaign guidelines, a retrospective analysis of 15 000 patients enrolled in the campaign, 55 referenced studies, and three meta-analyses.¹⁵ Each lacked scientific rigour and provided a weak basis for endorsing a performance measure (box 2).^{15 30 36 37} Other interventions in SEP-1, such as the need for a 30 mL/kg fluid challenge and serial lactate determinations, also failed to meet the National Quality Forum’s standards for scientific evidence.^{15 36}

Box 2. Critical analysis of evidence supporting central venous pressure and blood oxygen saturation.

Surviving Sepsis Campaign guideline (2008)

Reported evidence for recommending use of these measures was graded in the guidelines as weak (grade C)

Retrospective analysis of 15 000 patients enrolled in the Surviving Sepsis Campaign

• Data collection was not refereed

• Analysis lacked a control group

• Concluded no significant association between attaining targets and improved survival

Fifty five studies from the literature

• 17 of the 55 studies provided no data comparing the survival effects of a sepsis bundle of performance measures to controls

• 34 studies were observational studies

• Four were randomised controlled trials; the seminal study supporting the measures was a single centre, non-blinded study criticised for its statistical analysis and the financial conflicts of its primary author. The other three were conducted in China where medical practice and patient mix differ from the West

Three meta-analyses

• One concluded that the nine studies analysed were confounded and provided no convincing evidence supporting sepsis bundle use

• The other two included studies that were among the 55 problematic ones noted above, that did not assess patients with sepsis, or that included protocols not comparable with the performance measure under evaluation

Commercial interests

Edwards Lifesciences produces an oximetry catheter to continuously monitor central venous pressure and blood oxygen saturation and, therefore, stood to benefit from the measures being included in SEP-1. This company had paid or supported the research of the primary developer and two co-developers of SEP-1.^{30 37 38 39} AdvaMed is a medical technology trade association that promoted Edwards’s and other manufacturers’ devices.⁴⁰ Edwards’s chief executive officer of has been the chair of AdvaMed’s committee on payment and healthcare delivery and the chair and a member of AdvaMed’s board of directors.^{41 42} AdvaMed nominated its vice president of payment and healthcare delivery to the National Quality Forum’s steering committee reviewing SEP-1 (National Quality Form, personal communication, 2014), and this vice president was co-chair of the steering committee during SEP-1’s endorsement review.⁴³ We know of no evidence that any person or company intended to or effectively influenced the forum’s 2013 decision to endorse SEP-1 and its requirement for central venous pressure and blood oxygen saturation. But the commercial relations are not in accordance with the forum’s policy on potential conflicts, indicate potential undue commercial influence, and weaken SEP-1’s credibility (box 1).

Reaching a compromise

In April 2014, the ProCESS trial of 1351 patients from 31 hospitals reported that protocol based haemodynamic therapy using central venous pressure and blood oxygen saturation did not significantly improve sepsis survival.³¹ After public comment, and with two more trials of the measures still under way, the Centers for Medicare and Medicaid Services announced in August 2014 that Henry Ford Hospital should collaborate with other stakeholders to reach a compromise on inclusion of the measures in SEP-1.⁴⁴ The two other randomised controlled trials (ARISE and ProMISe) confirmed the absence of significant benefit for central venous pressure and blood oxygen saturation for routine septic shock management.³¹ Improving sepsis care over the past decade and selection bias have been postulated to explain why these three trials did not reproduce the findings of the original pivotal study.^{45 46}

The National Quality Forum subsequently endorsed changes to section F of SEP-1 based on the compromise reached between the measure developers and key stakeholders including the Society of Critical Care Medicine.⁴⁷ Central venous pressure and blood oxygen saturation measures were still included.

The revised SEP-1 said that a volume status and tissue perfusion assessment could be fulfilled if all five components of a focused exam were completed by a licensed independent practitioner (capillary refill, cardiopulmonary examination, peripheral examination, skin examination, vital signs) or if two of four measures were taken (central venous pressure, blood oxygen saturation, bedside cardiovascular ultrasonography, or fluid responsiveness using leg raise or fluid challenge).⁴⁸ A well intentioned provider might still consider central venous pressure and blood oxygen saturation as the best choices because the other options are either more expensive, infrequently used, or are not described or evaluated in standard adult sepsis guidelines. Continued inclusion of the measures in SEP-1 gives oximetry catheter manufacturers a powerful marketing tool to promote a non-beneficial device for routine sepsis management.⁴⁹

The primary developer of SEP-1 continued to receive financial support from Edwards.³⁸ Furthermore, two co-developers of SEP-1 with a past history of commercial relations with Edwards were both past presidents of the Society of Critical Care Medicine, an organisation with financial relationships with Edwards. One of the two SEP-1 co-developers was the society’s representative to the National Quality Forum.^{36 50 51 52} The Centers for Medicare and Medicaid Services notified hospitals that data collection for the revised SEP-1 would begin in fiscal year 2015.⁵³

Conclusions

The Centers for Medicare and Medicaid Services adopting SEP-1 will help make an unproven commercial product a national standard of care for the management of sepsis. Trade associations or other commercial entities and their representatives with the potential to profit, even if indirectly, from a particular endorsement should be excluded from that endorsement process. At present, such entities can participate in and vote at every stage of the National Quality Forum’s endorsement process. To avoid apparent or unseen conflicts of interest, the US Food and Drug Administration excludes commercial representatives from the approval of new drugs or devices. Given the influence that performance measures endorsed by the National Quality Forum have on national healthcare, this agency should adopt similar practices. If governmental agencies do not ensure that performance measures are based on sound scientific evidence and are safeguarded from potential commercial influences and even the perception of conflicts of interest, trust in these agencies will be undermined and patient care compromised.