The use of report cards and outcome measurements to improve the safety of surgical care: the American College of Surgeons National Surgical Quality Improvement Program

Abstract

Postoperative adverse events occur all too commonly and contribute greatly to our large and increasing healthcare costs. Surgeons, as well as hospitals, need to know their own outcomes in order to recognise areas that need improvement before they can work towards reducing complications. In the USA, the American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP) collects clinical data that provide benchmarks for providers and hospitals. This review summarises the history of ACS NSQIP and its components, and describes the evidence that feeding outcomes back to providers, along with real-time comparisons with other hospital rates, leads to quality improvement, better patient outcomes, cost savings and overall improved patient safety. The potential harms and limitations of the program are discussed.

The problem

Over 30 million operations are performed in the USA each year.¹ Postoperative adverse events occur frequently. Even in common, non-complex cases such as colectomy (250 000 cases annually), surgical site infections occur in 10% of patients.² These adverse events increase hospitalisation length and cost. For example, one study estimates that the cost of a surgical site infection is over US$27 000,³ while a urinary tract infection can cost from US$675 to US$2800.⁴ A single case of ventilator-associated pneumonia can add US$50 000 to the cost of an admission.^{5 6} Length of stay increases when complications occur, with an extra 3–11 days in hospital required for respiratory events.⁷ Payments are being reduced for some adverse events, such as central line infections and surgical site infections. The Affordable Care Act of 2012 established the Hospital Readmission Reduction Program and in turn the Centers for Medicare and Medicaid Services began implementing a process to reduce payments for certain 30-day readmissions.

The largest and best known intervention for measuring and reporting surgical outcomes in the USA is the American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP). Risk-adjusted postoperative outcomes reported by NSQIP provide benchmarks intended to spur local quality improvement efforts to produce better patient outcomes. This review assesses the evidence to determine whether ACS NSQIP has improved surgical care and patient safety.

Patient safety strategies

The multi-component ACS NSQIP in part grew out of efforts initiated by Veterans Affairs (VA) health system researchers and clinicians in the late 1980s. In response to concerns about high complication rates in VA hospitals, VA NSQIP was officially launched in 1994 to collect and report clinical variables and outcomes across all VA hospitals.^{8 9} Another factor contributing to ACS NSQIP was the success of programs like the New York State Cardiac Surgery Reporting System (CSRS) in the 1990s.¹⁰ Basic concepts of the New York State CSRS have been adopted by other states (California, Pennsylvania) and have spread across the USA through the efforts of the Society of Thoracic Surgery (STS) Registry, which incorporated public reporting. The measurement and public reporting of cardiac surgery outcomes has also spread to England.¹¹

With growing attention being paid to systematic reporting of surgical outcomes, non-VA hospitals became interested in applying the VA experience to their data reporting and quality improvement programs in the late 1990s. A pilot study in three civilian hospitals (University of Michigan, Emory University and University of Kentucky) demonstrated the feasibility of this initiative in the private sector.¹² Following this pilot, the ACS took the lead to expand efforts to a broader group of hospitals (14 sites) in 2001, and the formal ACS NSQIP began in 2004.¹³ An overview of the key aspects of the program are provided in figure 1.

Figure 1.

Key aspects of the American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP). *Surgeons selected the content, make decisions about changes in the way data are collected and analysed, and they provide oversight.

The intervention consists of several basic components. First, a surgeon champion (an individual staff member at the participating site) assumes a significant role in establishing and overseeing the participation of each site. Second, a surgical clinical reviewer (SCR) is trained to collect data on preoperative clinical variables and on 30-day outcomes. The third component is the development of risk-adjusted models of expected mortality and morbidity by procedure type. Fourth, data are analysed and presented back to the individual sites alongside masked data for the other sites. Data are provided for each of 21 morbidities (such as unplanned readmissions, urinary tract infections and pulmonary embolism) in addition to mortality. Sites are displayed as being high (worse than expected) or low outliers (better than expected) for each category of morbidity and for mortality. Lastly, the participating sites are encouraged to address and correct problem areas of clinical care.

Originally, ACS NSQIP reported observed-to-expected (O/E) 30-day mortality and morbidity ratios. Over the past 8 years, risk adjustment and modelling has evolved to improve the accuracy of the reported outcomes.¹⁴ Recently, the system has been adapted to better account for surgical case complexity by introducing a procedure mix adjustment. Additionally, ACS NSQIP no longer reports O/E ratios but instead provides logistical ORs using hierarchical modelling with shrinkage adjustment (figure 2). This allows for more stable estimates for small sample sizes, such as lower volume procedures or smaller hospitals. Additionally, this allows for more valid and accurate reporting of individual provider rates. Currently, both facility and surgeon data are collected. Individual facility results are provided routinely, but those for individual surgeons are provided upon request.

Figure 2.

Example reporting events by odds ratio using hierarchal modelling with shrinkage adjustment. Hospital specific bar plots. Each box shows the distribution of odds ratios for hospitals in the model; the top and bottom of each box give the smallest and largest values, and horizontal lines give the deciles. The point and vertical line within each box give the individual hospital's odds ratio and 99% CI. General and vascular surgery (GV) outcomes models are reported. DVT, deep venous thrombosis; PE, pulmonary embolism; ROR, return to operating room; SSI, surgical site infection; UTI, urinary tract infection. Adapted from Cohen, JACS, 2013.¹⁴

From reports generated by ACS NSQIP, institutions can see where they are a high outlier for certain procedures and identify where improvement is needed. An example of a de-identified report is provided in figure 2. Auditing by ACS NSQIP staff occurs randomly and for cause, that is if a site reported results too good to be true, and an audit for cause revealed upcoding of comorbidities and downcoding of complications or dramatic changes in results. The audits are also measures of data collection quality. Program audits reported low disagreement rates (between the auditor and site data collector) relatively early in program development (3.15% in 2005) and this rate has continued to drop (1.56% in 2008).¹⁵

While the responsibility for making changes remains with the individual sites, the administrative ACS NSQIP body provides support in terms of case report of successful interventions, provision of best practices, national meetings, and monthly supportive conferences calls with the surgeon champions and clinical reviewers. For example, ACS NSQIP staff will reach out to individual sites to assist them by connecting them with others who have made improvements in the area of concern or interest. Additionally, the annual ACS NSQIP meeting serves as a critical forum for sites to present and discuss a range of quality improvement efforts and successes.

Review processes

A Medline search up to November 2012 carried out using the search terms NSQIP and National Surgical Quality Improvement Project identified 169 studies. The review also included the Cochrane Registry and gray literature available on the ACS NSQIP website (http://www.acsnsqip.org/). Following title and abstract screening, the author selected any article that provided evidence for the benefits or harms of VA NSQIP or ACS NSQIP, or quantitative information describing how ACS NSQIP was implemented. There was no restriction on study design.

The author supplemented this literature review with interviews with the leadership and administrators in ACS NSQIP, as well as four surgeon champions. These surgeons were selected to represent different participants in the program: community, academic, leadership and a state-wide collaborative. The interviews explored weaknesses as well as strengths of the program.

Benefits and harms

Benefits

A VA NSQIP review of over 400 000 cases performed between 1991 and 1997 showed that 30-day mortality and morbidity rates for major surgery fell by 9% and 30%, respectively.¹⁶ Reductions in one postoperative complication alone, surgical pneumonia, are estimated to have saved the VA US$9.3 million annually, and the overall reduction in morbidity may have saved billions since the program was started.^{8 17 18}

Two published longitudinal studies reached divergent conclusions on the effects of reporting in ACS NSQIP. They first looked at changes from 2005 to 2007 in ACS NSQIP-participating sites (N=183) for all outcomes measured and surgical specialties using risk adjustment and accounting for hospital procedure volume.¹⁹ For the most recent period of 2006–2007, 118 hospitals participated long enough to produce clinically useful data. The authors found that 82% of hospitals had improved morbidity and 66% had improved mortality. The adjusted absolute difference in O/E ratio was −0.114 for morbidity and −0.174 for mortality (negative numbers indicate less morbidity and mortality). Similar results were seen when the researchers accounted for institutional volume. They also found that the number of high outliers (those with worse outcomes) decreased over time and the number of low outliers (those with better outcomes) increased. Institutions with high outliers were more likely to improve and had larger mean changes in outcomes. It was estimated that an average of 200–500 complications and 12–36 deaths may have been avoided.¹⁹

The other study compared ACS NSQIP to a private sector collaborative based at the University of Michigan Medical Center.²⁰ The Michigan Surgical Quality Collaborative (MSQC) includes 34, mainly community (68%) hospitals, unlike the ACS NSQIP-participating hospitals, which are primarily academic/teaching institutions. Sixteen MSQC hospitals were assessed over two time periods and compared to the 126 non-Michigan NSQIP hospitals over the same time periods. MSQC hospitals had a decrease in morbidity from 10.7% to 9.7% (9.0% reduction, p=0.002; OR 0.898) over 3 years, whereas morbidity did not change for the ACS NSQIP hospitals in either time period or between the periods (12.4%; OR 1.0).²⁰

The potential impact of participating in ACS NSQIP on complication rates and mortality has been reported by individual hospitals and collaboratives. Although improvements in morbidities have been large, mortality changes have ranged from none to modest. Mortality for most general and vascular surgical procedures is generally low, leaving little room for improvement. In contrast, improvements in morbidities are commonly reported. One local initiative showed that targeted efforts reduced respiratory complications and that after 7 months the rate of postoperative pneumonia had been reduced to zero from a peak monthly rate of 2%.²¹ Another state collaborative showed improvements in a range of postoperative events.²² However, most of the reports of improvement in single institutions or later collaboratives have been presented in forums other than peer-review publications.

At the July 2011 ACS NSQIP national meeting, 20 presentations reported reductions in morbidity following an intervention. In all these cases, ACS NSQIP data enabled the hospitals to target an area with worse-than-expected outcomes and to intervene, with resulting improvement (table 1). These presentations were selected as representative of successful quality improvement efforts that developed as direct responses to outcomes shown to be in need of improvement. These collective results demonstrate how varying sites and collaboratives worked to make specific changes to improve outcomes.

Table 1.

Example of interventions and changes in outcomes in ACS NSQIP hospitals/collaboratives

Hospital	Complication	Intervention	Outcome
Hershey Medical Center, Penn State35	19.3% SSI in patients with diabetes; 8% in patients without diabetes VTE 3.4% (2008)	Glucose control protocol VTE risk assessment and order set	Reduction of SSI O/E from 1.31 to 0.78 Reduction of VTE rate from 3.4% to 0.2% (2008–2009)
University of Virginia37	17.6% SSI (national average 8.1%) in colorectal resections, high BMI was a risk factor	Protocol for wound wicking for BMI >25 kg/m2, SCIP measures, glycaemic control	Reduction of SSI from 17.6% to 11.2% (36% reduction) (2003–2006)
Massachusetts General Hospital37	Vascular surgery morbidity O/E ratio 1.19, (99% CI 0.93 to 1.48) UTI rate 7.0% vs 4.7% (p<0.087)	Physician order entry templates, Foley catheter removal algorithm, silver-coated catheters for selected patients, identify procedures not requiring a catheter, educational campaign for clinicians	Reduction of UTI from 7.0% to 1.8% Morbidity O/E ratio decreased from 1.19 (99% CI 0.93 to 1.48) to 0.93 (99% CI 0.67 to 1.48) (76% reduction) (2003–2004)
Hospital A	Identified a rise in organ space infections	Standardised orders, proper antibiotic use, morbidity conference presentations, skin preparation changes	Organ space infection increase attributed to increased leak rates and identified surgical technique issues; improvements seen, but rate still high (2005–2010)
Hospital B	VTE 17.6%	Risk stratification, best practices, standardised orders	VTE decreased from 17.6 to 2.3%; O/E decreased from 1.88 to 1.05 (2006–2010)
Hospital C	Unplanned reintubation 3% (O/E 1.56) Ventilator>48 h 3.84% (O/E 1.71)	Tracking tool, risk assessment, improved pulmonary hygiene intervention	TBD
Hospital D	Ventilator use for >48 h 2.24% (O/E 1.7)	Tracking tool, standardised orders, patient education	Ventilator use for >48 h decreased from 2.24% to 1.19% (O/E 1.7 to 0.83) (2008–2010)
Hospital E	Overall orthopaedic DVT rate 3.1% Knee arthroplasty DVT rate 10.1%	Identified variations in DVT prophylaxis practice, surgeon-specific review, standardised care	Reduction of overall orthopaedic DVT rate from 3.1% to 1.1% Reduction of knee arthroplasty DVT rate from 10.1% to 1.6% (2008–2010)

Hospitals A–E are representative examples taken from the ACS NSQIP data portal website, accessed 13 December 2011. Reprinted by permission of American College of Surgeons NSQIP.

ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Project; BMI, body mass index; DVT, deep venous thrombosis; SCIP, Surgical Care Improvement Project; SSI, surgical site infection; TBD, to be determined; UTI, urinary tract infection; VTE, venous thromboembolism.

Almost all these studies have a pre–post design and therefore have all the accompanying limitations, including regression to the mean. However, the results are unlikely to be explained solely by regression to the mean, primarily because of the size of the effect and the post-intervention result. Additionally, these changes occur in response to directed efforts for particular areas, such as decreasing deep venous thrombosis (DVT) rates or reintubation rates. Furthermore, Hall et al²³ demonstrated that both bad (high outlier) and good (low outlier) hospitals improve. Taking into consideration the large body of pre-post work, which shows repeatedly each year of the program, support that directed efforts were largely responsible for the reported improvements.

The way NSQIP improves care is multifaceted. To improve care and reduce complications, surgeons must know the outcomes of their own procedures. The data must be of high quality and reliable, and risk adjustment must be adequate to allay concerns about comparing ‘apples to oranges’. This comparison allows surgeons and hospitals to see how they compare in terms of outcomes, which promotes accountability and stimulates work to correct the problems. Most sites (59% of those surveyed) were unaware of their hospital's adverse event rates, let alone how they compared to other hospitals, until after they enrolled in ACS NSQIP.³

A particular feature of the program is its use of detailed clinical data collected from the medical records. A study comparing administrative and claims data collected by the University Health System Consortium (UHC) program showed that ACS NSQIP identified a greater number of complications (61%) than UHC, including 97% more surgical site infections and 100% more urinary tract infections.²⁴

Furthermore, ACS NSQIP identifies adverse events following discharge. Studies using ACS NSQIP show that more than 50% of complications occur after discharge. For colectomies, 45% of deep surgical site infections, 39% of organ space infections and 28% of DVT occur after patients have left hospital.²⁵ Identifying complications that occur outside the hospital is the prerequisite first step to developing changes in care to help prevent them.¹⁹

Harms

Few published studies have assessed the potential and actual harms of this program, and most are speculative. A primary concern has been that surgeons will avoid high-risk cases for fear of adversely affecting their O/E outcomes assessments. This issue was raised early in the process of implementing report cards when anecdotal evidence appeared to suggest that as the result of implementing the New York CSRS, high-risk coronary artery bypass graft (CABG) patients were being diverted instead to the Cleveland Clinic.²⁶ However, subsequent and more comprehensive analyses could not document any systematic exclusion of high-risk patients from CABG operations, and showed that, on the contrary, the severity of illness and comorbidities of operated patients has increased over the years.^{27 28} The longitudinal ACS NSQIP study also supported this finding, showing that the risk profile and illness severity for surgical patients has increased over time.¹⁹ Another concern is that the outcomes for outpatient cases or for a hospital or surgeon who performs a small volume of procedures might need longer follow-up, possibly for more than a year, to accurately assess quality.²⁹ The question has been raised that surgeons could alter treatment plans for patients based on individual operative risk rather than giving the patient the option of a procedure with a potentially better long-term functional outcome. A theoretical example would be in vascular surgery, where a high-risk patient eligible for a distal bypass would be recommended an amputation instead.

Implementation considerations and costs

Implementation context

The program requirements include site administrative support, a surgeon champion, and participation in a series of conference calls and the national ACS NSQIP meeting. Data reporting is mandated to follow particular rules, such as accrual of particular data and 30-day follow-up information. ACS NSQIP personnel perform audits to help maintain data quality. For small hospitals, the effort and cost may be less than for larger facilities, depending on the volume of cases.

ACS NSQIP has been implemented in a variety of settings including large academic hospitals, smaller community hospitals and large and small state-wide consortia. It soon became apparent that a variety of program models were needed to accommodate differing clinical volumes. Program options vary in terms of number of variables collected, surgical specialty, if procedures are specifically targeted, and case sampling required. Currently, more than 500 sites are enrolled in ACS NSQIP, which represents roughly 10% of the almost 4500 hospitals in the USA. The distribution of the more than 525 sites that reported clinically useful data is shown in the map in figure 3.

Figure 3.

Geographical distribution of American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP) participating sites. Number of sites by state, region and country included in the January 2013 NSQIP semi-annual report. Reprinted by permission of ACS NSQIP.

Overall, 49% of participating sites are teaching or academic centres. The majority of these hospitals are high volume, as only 3% perform less than 100 procedures per year, 7% perform 100–299 procedures per year, 43% perform 300–499 procedures per year, and 47% perform more than 500 procedures per year. This skewed distribution of hospital size means that the 10% of hospitals participating in ACS NSQIP represent 32% of the procedures performed.³⁰ Certain complex procedures are captured at an even higher rate, for example, 57% of oesophagectomies and 53% of pancreatectomies billed to Medicare are performed at ACS NSQIP sites (table 2).

Table 2.

Percentage of Medicare surgical cases covered by ACS NSQIP

Procedure|	MC cases in NSQIP	MC cases not in NSQIP	Total MC cases	Percentage covered by NSQIP
Oesophagectomy	1158	875	2033	57.0%
Cystectomy	3346	4501	7847	42.6%
Abdominal aortic aneurysm repair	3762	6448	10 210	36.8%
Pancreatectomy	3901	3399	7300	53.4%
Colectomy	32 444	103 056	135 500	23.9%
Proctectomy	6745	15 767	22 512	30.0%
Aortoiliac bypass	2255	4974	7229	31.2%
Lower extremity bypass	12 203	30 100	42 303	28.8%
Liver resection	2465	2201	4666	52.8%
Hip fracture repair	40 030	151 140	191 170	20.9%
Abdominoplasty	1058	1829	2887	36.6%
Lung resection	16 065	27 391	43 456	37.0%
Endovascular abdominal aortic aneurysm repair	8944	17 324	26 268	34.0%
Nephrectomy	9727	16 375	26 102	37.3%
Hysterectomy	17 954	45 108	63 062	28.5%
Total hip arthroplasty	56 700	195 528	252 228	22.5%
Laminectomy	60 650	154 858	215 508	28.1%
Transurethral resection of the prostate	11 345	42 928	54 273	20.9%
Ventral hernia	19 360	57 735	77 095	25.1%
Carotid endarterectomy	20 588	59 710	80 298	25.6%
Total knee arthroplasty	72 916	279 642	352 558	20.7%
Prostatectomy	10 677	18 808	29 485	36.2%
Breast reconstruction	455	700	1155	39.4%
Appendectomy	8802	31 635	40 437	21.8%
Thyroid	5358	12 598	17 956	29.8%
Gastrectomy	3782	7382	11 164	33.9%
Carotid stent	3648	7883	11 531	31.6%
Small bowel resection	10 784	30 836	41 620	25.9%
Mastectomy	6417	21 378	27 795	23.1%
Cholecystectomy	29 386	117 327	146 713	20.0%
Total				32.0%

ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Project; MC, Medicare.

Collaboratives are a main feature of ACS NSQIP and have taken many different forms (some are a handful of geographically close hospitals while others are all hospitals in a state) that work as a team to implement the program and quality improvement. They also can represent a disease or patient population and so members of a collaborative need not be geographically close together. In addition to providing intellectual and practical support to each other for quality improvement initiatives, they provide a collective voice for bargaining with potential sources of funding. One reported approach is for the main insurer for the hospitals in the collaborative to pay 50% of the cost of the program over a set number of years. Sometimes an option to renew the financial support is given if certain milestones are met. Some payors have judged there is a business case for helping support ACS NSQIP participation because of perceived cost savings. Table 3 shows the current list of active collaboratives.

Table 3.

List of ACS NSQIP collaboratives including type, number of sites and payor

Group	Type	Number of sites	Payor involvement
Canadian National Surgical Quality Improvement Collaborative (CAN-NSQIP)	Regional	6	Canadian health authorities
Connecticut Surgical Quality Coalition (CTSQC)	Regional	5	None at this time
Department of Defense/TRICARE	System-wide	16	Department of Defense/TRICARE
Florida Surgical Care Initiative (FSCI)	Regional	63	BlueCross BlueShield of Florida
Fraser Health Systems (Canada)	System-wide	3	Fraser Health Authority
Illinois Surgical Quality Improvement Collaborative (ISQIC)	Regional	12	None at this time
Kaiser Permanente Northern California Regional NSQIP Collaborative (KPNCRNC)	System-wide	21	Kaiser Permanente Northern California
Kaiser Permanente Southern California Regional NSQIP Collaborative (KPNCRNC)	System-wide	8	Kaiser Permanente Southern California
MaineHealth Collaborative	System-wide	6	MaineHealth
Mayo Clinic Surgical Quality Consortium (MCSQC)	System-wide	5	Mayo Clinic
Northern California Surgical Quality Collaborative (NCSQC)	Regional	4	None at this time
Nebraska Collaborative	Regional	2	BlueCross BlueShield of Nebraska
Oregon NSQIP Consortia	Regional	8	None at this time
Pennsylvania NSQIP Consortia	Regional	10	None at this time
Partners HealthCare	System-wide	5	BlueCross BlueShield of Massachusetts
Surgical Quality Action Network—British Columbia, Canada (SQAN)	Regional	21	BC Patient Safety and Quality Council
Tennessee Surgical Quality Collaborative (TSQC)	Regional	10	BlueCross BlueShield of Tennessee Health Foundation
Upstate New York Surgical Quality Initiative	Regional	7	Excellus
ACS NSQIP Colectomy Collaborative	Virtual	36	None at this time
ACS NSQIP Glucose Control Collaborative (Pending)	Virtual	4	None at this time
ACS NSQIP Rural Collaborative (Pending)	Virtual	5	None at this time
ACS NSQIP Residency Training Collaborative (Pending)	Virtual	TBD	None at this time
Indiana Collaborative (Pending)	Regional	7	None at this time
Maryland Collaborative (Pending)	Regional	3	None at this time
Texas Collaborative (Pending)	Regional	16	None at this time
Virginia Collaborative (Pending)	Regional	11	None at this time
Wisconsin Collaborative (Pending)	Regional	6	None at this time

Adapted from ACS NSQIP Annual Meeting, July 2011. Reprinted by permission of American College of Surgeons NSQIP.

ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Project; TBD, to be determined.

A pilot paediatric ACS NSQIP collaborative collects data for patients under 18 years of age.^{31 32} Variables have been modified for paediatric surgery practices and needs.

There are many examples of different collaboratives, along with unique challenges facing different hospital types. The hospitals proposing to develop a collaborative negotiate as a group for financial support from a variety of sources, shape the program for their own needs, and work together to make quality improvement changes. Two examples, one of a community of hospitals starting a small state-wide collaborative (Tennessee Surgical Quality Collaborative, TSQC) and another where state-wide participation in a collaborative was developed (Florida Surgical Care Initiative, FSCI), are described in detail below.

In 2004, after being introduced to the recently started ACS NSQIP at the national ACS meeting, a member of a community hospital in Tennessee initiated a process that resulted in a collaborative that included hospitals, payors and the Tennessee Hospital Association. The collaborative would be controlled by a leadership committee that comprised four surgeons appointed by the local society chapter, along with hospital CEOs and a member of the Tennessee Hospital Association. The proposal included funding for the participation of eight hospitals, estimated to be US$2 550 000 for 3 years. The money covered half the expense of the SCR, salary support for the surgeon champions, and administrative costs. This example highlights many of the key components for building a successful program—surgical leaders taking a role, supportive administration, and collaboration with other hospitals.

A strikingly different collaborative was set up in Florida. The Florida Hospital Association (FHA) was aware of the high surgical mortality demonstrated by the Dartmouth Atlas project (which identified disparities in access to and utilisation of health care) in their state.³³ The FHA, along with the payor, BlueCross BlueShiled, collaborated to generate a financial incentive for hospital participation. A new version of the program collecting only four outcomes was developed, thus lowering costs. Currently, 64 hospitals are participating in the FSCI and the participation of 39 more is pending. This example demonstrates additional features that help encourage participation: individuals at the state level and hospital administration taking a lead, a flexible program designed to fit the needs of the collaborative, and the role of the local payor in incentivising hospital participation.

Costs

The costs of participation vary depending on what type of program the hospital joins. The annual administrative fee varies by hospital size and level of participation, salary of the SCR, and optional bonus payments to support the surgical champion or quality improvement team. This fee ranges from US$10 000 (rural and hospitals that deal with <2000 cases/year) to US$25 000 (>2000 cases). Hospitals can lower their costs by participating in a collaborative.

The salary for the SCR comprises the bulk of the expense of participation. Previously, the clinical reviewer had to be a registered nurse (RN), but because of issues such as nursing shortages, individuals such as licensed vocational nurses, with medical training but without advanced nursing degrees, have been successfully employed as SCRs. All reviewers must pass a credentialing examination annually. Their expenses will vary based on experience, level of training and region, and range from around US$40 000 per year to US$100 000 for an experienced RN. Many hospitals suggest that paying for a surgeon champion (an amount such as US$5000) is helpful for increasing their involvement, although a recent survey of surgical champions (109 respondents) found that 72.5% did not receive salary support compensation.³⁴

The highest total cost of participation is estimated to be US$135 000 annually; however, this estimate is for a large hospital that hires an RN as the reviewer.^{35 36} Many participating hospitals may pay less as they have lower volumes of patients and therefore decide to participate in a smaller program.

Since the overarching goal of ACS NSQIP is to reduce complications, which are costly, the business case for participating is that the expense of the program translates into savings for the hospital. Examples of such savings reported by NSQIP sites are shown in table 4. Pre–post data without control groups are shown and so inference of a causal relationship is limited by the study design.

Table 4.

Published examples of changes in complications and costs following participation in ACS NSQIP

Hospital	Complication reduction	Savings
Surrey Memorial Hospital32	Reduced SSI over four years: from 13%, to 10%, to 7.5%, to 7.2%	US$2.54 million in savings
Henry Ford Hospital36	Reduced LOS by 1.54 days over 4 years for general surgery, vascular and colorectal procedures	US$2 million in annual savings (increased billing by US$2.25 million/year as underbillings were identified)
VA8	Surgical pneumonia alone	US$9.3 million in savings annually
University of Michigan Medical Center5	Respiratory complication	US$51 409 per event; a reduction of two such complications per year pays for participation
Hershey Medical Center, Penn State33	Additional cost attributable to a postoperative complication is US$16 371	Avoiding one postoperative complication equals a cost saving of US$9052

ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Project; LOS, length of stay; VA, Veterans Affairs.

One cost-effectiveness study of ACS NSQIP has been published. Costs and outcomes for 2229 general and vascular surgery cases at one large academic study were assessed. The study compared two time intervals - one 6 months and the other 1 year from inception of the program. Postoperative events declined over time, from 17 to 13 percent. The incremental costs were $832 and $266 per patient for the two time periods, meaning the cost per patient of the program declined after the first 6 months of participation. The incremental cost-effectiveness ratio to avoid 1 postoperative event was $25,471 in the first 6 months, declining to $7,319 in the second time period, meaning that the longer the institution participated in the program, the more cost-effective the program became.³⁷

Discussion

Although no randomised trials have assessed the use of outcomes measurements and reporting in surgery, the strength of the evidence that doing so improves operative mortality and morbidity is moderate or even high, given the strong theoretical rationale why it should work, the evidence that outcome reporting has likely improved surgical outcomes in other settings (eg, the New York state CSRS), the numerous reports from ACS NSQIP sites describing the implementation of quality and safety initiatives following identification of high outlier status, and the ensuing, sometimes dramatic, improvements in those outcomes. There is a great deal of experience on how to implement ACS NSQIP as it has been rolled out in more than 400 hospitals. Some of the key components of ACS NSQIP (collecting complications data, sharing models of O/E results, multi-site data collection systems across institutions that provide results back to the sites for benchmarking, contexts for learning and sharing tools that appear to be effective across sites) are similar to those of other successful patient safety practices such as the Michigan Keystone ICU Project to reduce catheter-related bloodstream infections.³⁸ Despite ACS NSQIP and the Keystone ICU Project having started with different original ‘interventions’ (the feedback of procedure-specific surgical outcome data to surgeons and a checklist of processes to reduce infections), the fact that current versions of the interventions include many similar components suggests the implementation of certain types of practices across hospitals is generalisable.

ACS NSQIP provides hospitals and providers with usable clinical data that are otherwise not available to them. Currently, all hospitals use administrative data to some degree to assess quality through the CMS Hospital Compare program or the Surgical Care Improvement Project (SCIP). These data lack clinical information and are limited by the variables reported for claims. More importantly, the correlation between administrative data and actual complications or diagnoses is inadequate. ACS NSQIP uses detailed clinical data to highlight areas where improvements are needed.

The greatest benefit has been seen in the larger hospitals in the procedures with higher complication rates. Whether the above improvements will transfer to low-risk but common procedures, such as outpatient procedures, is unclear. Most of the early adopters have been large academically affiliated hospitals. How successfully and widely the program can be implemented at smaller hospitals remains to be seen. ACS NSQIP has been flexible in terms of changing to fit the needs of those participating. The program specifics have been adapted for variable hospital sizes and large versus small collaboratives, and the program has provided vastly different modules for specific procedures, and even a new program for a specific surgical population—ACS NSQIP Pediatric for paediatric surgery reporting.

NSQIP was developed specifically for surgery. Like other surgical outcomes reporting systems, the conceptual model behind ACS NSQIP works best in situations where outcomes are measureable within a short time frame after the relevant care has been delivered, and there are reasonable means to adjust for case-mix differences. This model may not generalise to all types of health care, such as primary care, chronic care etc, but may translate well to those sharing similar clinical properties, like treatment provided in intensive care units.

A limitation of ACS NSQIP, or any outcomes-based quality improvement program, is that knowing outcomes does not necessarily provide the answer to producing better outcomes, rather, it requires the unit to know itself and identify the defect. ACS NSQIP does provide educational tools, such as guidelines and best practices, but it still requires leadership at the local level to lead the charge. Lastly, as process improvement is needed to sustain better outcomes, it is also important to measure and benchmark process compliance, which ACS NSQIP does not currently do.