Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Oct 24.
Published in final edited form as: Surgery. 2011 Nov 30;151(2):146–152. doi: 10.1016/j.surg.2011.08.015

Creating a learning healthcare system in surgery: Washington State’s Surgical Care and Outcomes Assessment Program (SCOAP) at 5 years

Steve Kwon a, Michael Florence b, Peter Grigas c, Marc Horton b, Karen Horvath a, Morrie Johnson d, Gregory Jurkovich e, Wendy Klamp a, Kristin Peterson f, Terence Quigley g, William Raum h, Terry Rogers i, Richard Thirlby j, Ellen T Farrokhi k, David R Flum a; For the SCOAP Collaborative, The writing group for the SCOAP Collaborative
PMCID: PMC4208432  NIHMSID: NIHMS636059  PMID: 22129638

Abstract

There are increasing efforts towards improving the quality and safety of surgical care while decreasing the costs. In Washington state, there has been a regional and unique approach to surgical quality improvement. The development of the Surgical Care and Outcomes Assessment Program (SCOAP) was first described 5 years ago. SCOAP is a peer-to-peer collaborative that engages surgeons to determine the many process of care metrics that go into a “perfect” operation, track on risk adjusted outcomes that are specific to a given operation, and create interventions to correct under performance in both the use of these process measures and outcomes. SCOAP is a thematic departure from report card oriented QI. SCOAP builds off the collaboration and trust of the surgical community and strives for quality improvement by having peers change behaviors of one another. We provide, here, the progress of the SCOAP initiative and highlight its achievements and challenges.

As described by the Institute of Medicine,1 building a “learning healthcare system” will improve care delivery by helping the experiences of other clinicians, patients, and informed stakeholders influence medical care received by any given individual. A learning system would parallel the experience in aviation where a best practice or safety provision identified in a single aircraft or flight influences service delivery for all similar aircrafts across the country, harnessing the experiences of one to help many. Developing such a system is critical in surgery and interventional care given the variation in quality, safety, appropriateness, and cost and the increasing complexity of our patients and the technology available to care for them.2–4

This variation has been the increasing focus of government-led quality improvement (QI) initiatives such as the Surgical Care Improvement Project (SCIP).5 SCIP rewards hospitals for reporting on the use of 9 quality-of-care metrics, increasing their use if not necessarily improving outcomes. Stulberg et al6 recently found that adherence to individual SCIP metrics for infection prevention was not associated with decreased postoperative infections, and critics have faulted this “metric-by-metric” approach to QI as not being a particularly efficient or effective path to significantly reducing variability. Government- and payer-directed QI programs are also viewed as punitive and create a defensive, “responder mentality” among clinicians.7 For example, in 2013, Medicare’s Value Based Purchasing System converts the incentive to participate in SCIP into a penalty for not participating. In states with public reporting of surgeon and hospital rates of mortality, a “shame-based” approach to addressing quality has been established resulting in a mix of intended and sometimes unintended results.8

Alternatives are 2 programs that surgeons have developed: The American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP)9 and the Washington State Surgical Care and Outcomes Assessment Program (SCOAP). The ACS-NSQIP is a “report card” system that provides hospitals with risk-adjusted rates of surgical mortality and morbidity. In its current iteration NSQIP provides risk-adjusted, 30-day aggregate morbidity and mortality outcomes (mostly for general and vascular surgery) with an observed/expected (O/E) ratio. An optional, version 3 of NSQIP is now available and includes procedure-specific covariates, process-of-care measures, and outcomes.9 Worst (and best) performing hospitals are identified as outliers in the hopes that surgeons and QI leaders at those hospitals will look to find out what types of operations and what elements of their performance are driving their outlier status. There has been a considerable effort by ACS to increase participation and traction for the program, especially because in some states insurers have started paying for its implementation. Approximately 5% of US hospitals are now enrolled, with most tending to be academic medical centers or larger centers that are capable of absorbing the costs of the abstractors and program fees.

Clinicians in Washington State developed a different approach, developing the backbone of a “learning” healthcare system in surgery that is driving broad process improvement, and reducing variability in process of care, adverse outcomes, and appropriateness of care and healthcare expenditures. Washington State’s SCOAP (available: www.SCOAP.org) was first described in this journal 5 years ago.10 SCOAP’s approach to QI recognizes that there are too many components involved in high-quality surgical care for a “piece-by-piece” approach like SCIP and that clinicians do not necessarily have the time to drill down from global metrics of O/E to find out where there are opportunities for improvement. SCOAP surgeons recognized that there is too much nuance between patients and settings to count on overall O/E ratios to understand and change clinical practice and that just because your hospital is not an outlier does not mean that things cannot be improved. SCOAP is a collaborative of clinicians that engages its members by having them determine the many process of care metrics that go into a “perfect” procedure, track risk-adjusted outcomes that are specific to a given operation, and create interventions to correct underperformance in both process measures and outcomes. SCOAP includes >50 process-of-care measures, some that have evidence links with risk-adjusted outcomes and others that have emerged through a consensus process. Another 50 measures explore a range of emerging practices, drugs, devices, and process-of-care measures that SCOAP tests to determine their value. SCOAP spans across clinical disciplines bringing together all the clinicians who perform procedures or whose work touches on surgery (including radiology, pathology, and interventional radiology). Programs like SCOAP are a thematic departure from “surgeon-only,” report card–oriented QI and aim to drive improvement by having multidisciplinary peers change each other’s behaviors, building off the fellowship of the clinical community. This article provides progress on the SCOAP initiative and to highlight its achievements and challenges.

WHAT IS SCOAP?

SCOAP is a grassroots, voluntary, QI collaborative focused on surveillance and benchmarking of consecutive surgical procedures and interventional care, coupled with activities to address underperformance. In Washington State, most hospitals claim limited resources for QI activities and are low-volume centers where O/E mortality rates are not relevant metrics and where surgeons are looking for practical, actionable, low-cost tools that can help them deliver optimal care. SCOAP was designed by surgeons to deliver not just risk-adjusted morbidity and mortality rates, but a combination of procedure-specific outcomes particularly relevant to surgeons and their patients and process-of-care measures associated with optimal care. SCOAP’s process-of-care measures go far beyond those required by payer-based programs. They are developed by a range of community practice and academic surgeons based on a combination of experience, common sense, and developing evidence. In SCOAP, surgeons have defined a set of quality metrics that they expect to have direct links to optimal surgical care (ie, boosting the accuracy of diagnostic imaging used in patients with presumed appendicitis, avoidance of blood transfusions unless the hemoglobin is <7 g/dL, nutritional supplements for patients undergoing elective surgery who are malnourished, and appropriate use of neoadjuvant therapy for rectal cancers). Beyond surveillance of process and outcome performance, SCOAP is a collaborative of engaged clinicians from across clinical domains encouraging coordinated responses (educational interventions, preprinted orders, operating room checklists, etc) to address underperformance and then reinforcing behavior change by tracking performance improvement. SCOAP is also serving as a forum for comparative research and to determine whether there really is a link between process of care and outcome.

In distinction to a center of excellence model, SCOAP treats interventional care quality as a public health issue and works to improve the quality at every hospital. Because it is surgeon led and protected from medicolegal discovery, this peer-based collaborative has been a positive force for improving surgical quality and safety as well as surgical culture. Collaboratives like SCOAP, the Michigan NSQIP-based QI program,11 the Northern New England Collaborative,12 Washington State’s COAP,13 and efforts originally led by Hiram Polk in Kentucky14,15 have shown their value both in terms of improved outcomes and in changing the culture of surgeons within these peer-to-peer collaboratives.

SCOAP has 2 components—a surveillance system gathering data on patient risk-adjusted outcomes and processes of care of consecutive procedures, and an active “correction” function that engages surgeons to address lapses in care delivery. The surveillance system relies on information technology infrastructures of varying sophistication [from paper based records to fully integrated electronic medical record (EMR)] and joins surgeons at hospitals from all over the state, in rural and urban environments, in a data-sharing/feedback network. The corrective function of SCOAP works through education and peer support/pressure. Through e-newsletters and regional/statewide meetings clinicians participate in frequent peer-led educational and interventional activities aimed at creating behavior change around metrics, sharing best practices, and improving outcomes. SCOAP has also been continuously assessing the processes and outcomes associated with emerging procedures (ie, laparoscopic gastric sleeve, endovascular procedures of the lower leg) and helping to develop the next generation of process-of-care measures. SCOAP’s clinician-led management committee has built in flexibility in the data points that are being evaluated so it can include variables that describe innovations in the field and eliminate metrics without major interruptions in the database architecture.

PROGRESS TO DATE

SCOAP started in 2006 as a collaborative of 10 hospitals and as of July 2011 includes 60 out of 65 hospitals that perform at least some medium risk surgical care, defined by having ≥2 colon resections per year (representing >90% of surgical care in the state). SCOAP’s original procedures included appendectomy, bariatric surgery, and colorectal operations. SCOAP grew to include vascular surgery and interventional radiology procedures, pediatric surgery, and modules in spine, gynecology high-risk cancer procedures (esophagectomy, pancreatectomy, and hepatectomy), and outpatient (hernia repairs, cholecystectomy, and breast) procedures and urology are now being incorporated. Progress in enrolling hospital into this voluntary initiative has been facilitated by peer-to-peer networking, a public interest campaign, and the influence of stakeholders such as large, self-insured employers. SCOAP has focused on creating a value proposition for surgeons and hospitals to join. The SCOAP components that are reported by surgeons and hospital leadership as “value added” include providing actionable data in a timely fashion rather than a simple “report card” of mortality, allowing smaller hospitals and surgeons to demonstrate that they deliver highest quality surgical care even if they have smaller procedural volumes, convincing state legislators to not require surgeon/hospital reporting of outcomes using administrative data, linking of surgeons into a peer network and helping to reduce professional isolation for colleagues in small towns, meeting the requirements of the American Board of Surgery Maintenance of Certification Part IV program, and helping surgeons direct local QI activities to issues they care about and are of importance to their patients. Membership costs are quite low and a well-defined data dictionary allows for high-fidelity, audited data with lower level training for abstractors.

In its first 5 years SCOAP has focused energy not just on tracking variability but in creating an active response to reduce that variability (Fig 1, A and B). Surgeons in all hospitals now participate in the use of standardized orders/templates and a SCOAP OR checklist (available: www.SCOAPchecklist.org) that addresses many of the areas of underperformance (eg, best management of patients with diabetes while under anesthesia). “Checklisting” is one part of creating an active response to pathologic variability. In other examples of decreasing gaps in variabilities of surgical care practices, SCOAP has reduced the rates of negative appendectomy by encouraging the use of preoperative imaging (benchmarked for accuracy) among high risk patients (Fig 2).16 Improvements in intraoperative leak testing rates (Fig 3) and glycemic control interventions among diabetic patients undergoing elective colorectal operations (Fig 4) have been striking and not surprising, adverse events have declined with participation in SCOAP (Figs 1, B, 5, and 6). Participation in SCOAP also has led to significant cost savings. Since SCOAP began, it has literally “bent” the cost curve by helping hospitals avoid costly complications (Fig 7). The Table demonstrates a sampling of the metrics SCOAP examines for appendectomies, bariatric, and colorectal operations. A full list of metrics, data definitions, and risk adjustment schemes and data tools are available online (available: www.scoap.org).

Fig. 1.

Fig. 1

Open in a new tab

Variability in reoperative complications in Washington State hospitals (based on administrative data) with gray points representing those hospitals that eventually joined SCOAP (A) from 2000 to 2003 before SCOAP started (B) from 2008 to 2009, after SCOAP.

Fig. 2.

Fig. 2

Open in a new tab

Rates of preoperative imaging (ultrasonography or computed tomography) to diagnose appendicitis among women of childbearing age, by quarters of a hospital’s SCOAP participation.

Fig. 3.

Fig. 3

Open in a new tab

Rates of anastomotic leak testing in elective colorectal operations, by quarters of a hospital’s SCOAP participation.

Fig. 4.

Fig. 4

Open in a new tab

Rates of blood glucose checks in diabetic patients undergoing elective colorectal operations, by quarters of a hospital’s SCOAP participation.

Fig. 5.

Fig. 5

Open in a new tab

Negative appendectomy rates, by calendar quarters.

Fig. 6.

Fig. 6

Open in a new tab

Rates of operative complication in elective colorectal operations in sites (n = 6) that eventually joined SCOAP and had >50 operations per year in 2003 and in 2009.

Fig. 7.

Fig. 7

Open in a new tab

Average cost per case for appendectomy, colorectal, and bariatric operations, by calendar year. Total cost savings in 2009 is noted.

Table.

Examples of process of care and outcome measures captured in SCOAP

Process of care Outcome
Use of advanced diagnostic imaging (ultrasonography or computed tomography) for patients undergoing appendectomy Rate of negative appendectomy
Emergency room or urgent clinic visit within 1 week among the perforated appendices (exploratory)
Intraoperative leak testing in colorectal surgery Operative re-intervention (unadjusted and risk adjusted)
Use of staple line reinforcement in bariatric surgery (exploratory)
Normothermia maintained during operation Wound opened and/or antibiotic for infection
Use of opioid antagonist for ileus prevention (exploratory) Median postoperative day of first feeding and mean length of stay
β blockers appropriately continued Postoperative myocardial infarction
Avoiding elective colorectal resection among patients with albumin <3.0 g/dL Any postoperative intervention
Use of a preoperative nutritional intervention in appropriate risk patients (exploratory)
Receipt of postprocedure deep venous thrombosis chemoprophylaxis for highest risk patients Venous thromboembolism treated
Rate of transfusion among elective procedures and transfusions with hemoglobin <7 g/dL Postoperative respiratory failure (measured by mechanical ventilation free status)
Days with a urinary catheter
“Walking epidural”: In patients with epidurals, the “freedom” from urinary catheters after 48 hours (exploratory) Urinary tract infection
Open in a new tab

For a full list, see SCOAP data dictionary (available: www.scoap.org). Some are considered exploratory (nonbenchmarked as we determine their relationship to outcomes), whereas others are benchmarked against the collaborative’s “top 10%” performers.

As a learning healthcare system, SCOAP is an effective platform for developing the next generation of QI metrics and for comparative effectiveness research (CER). In October 2010, SCOAP was awarded an $11.7 million grant from Agency for Healthcare Research and Quality to leverage the program’s QI activities into a CER Translation Network (CERTN) that will be used to address CER questions across a range of clinical arenas. CER focuses on the impact of different treatment strategies in the “real world” rather than the typical research environment. SCOAP hospitals look to research funders like “the real world,” not just large medical centers, but inclusive of all types of hospitals, all members of “priority populations” for research funding agencies (eg, pediatric and elderly populations, minority groups, and those with comorbid conditions) as well as subjects who may not otherwise participate in clinical research (ie, lower socioeconomic status, lives far away from larger medical centers). Through CERTN, SCOAP data are coupled to records from ambulatory surgical centers, statewide payer billing and pharmacy systems, the state’s vital status registry, and a postdischarge function and quality-of-life survey assessment center. This data linking creates a unique longitudinal research record, incorporating almost all relevant data streams related to operative care and outcomes, and creating a research record of events and outcomes. CERTN is run through the University of Washington’s Surgical Outcomes Research Center within the UW Centers for Comparative and Health System Effectiveness.

CHALLENGES FOR SCOAP

Growing interest from surgeons and hospitals beyond Washington State to join SCOAP or create SCOAP-like activities highlights the opportunities for real clinician-led QI, but the barriers to these activities should be noted. Regional activities like SCOAP are challenging to develop and sustain. Linking clinicians and hospitals across regions requires a sense of community among these groups that may not naturally exist, and in many geographic areas these relationships have been strained through competition and other forces. Reconnecting as a clinical community and developing trust and mutual interest requires genuine engagement, leveling of hierarchies, and some bridge building. Although, in the future, incentives for activities like SCOAP may become aligned, there is currently no financial incentive for the volunteered time, team building, and development work that are a component of these initiatives and better performance through collaborative work is not specifically reimbursed. In some regions, the public health importance of an activity like SCOAP may not be enough to overcome the lack of incentives unless a large payer with dominance in the marketplace compels this activity. When a large payer does step in, these payer-led programs often become pay-for-performance initiatives or the payers demand access to underperformers. Creating “winners and losers” is a potential for initiatives with heavy payer involvement and may make the very hospitals that could most benefit from such initiatives less likely to join them until they become mandatory.

Another barrier involves the limits of EMRs. Health systems with better information technology can accomplish surveillance without as much of the chart abstraction that is required at hospitals without integrated and more functional EMRs. The reality of EMRs at most hospitals is a confusing maze of manual data aggregation and individual teams building their own data repositories and ad hoc data sets. This creates high operational costs, multiple solutions with redundant functionality, uncertainty about data validity, and low reusability of the data. A crucial challenge is bringing together structured data (data that reside in transaction systems such as clinical information systems and financial systems) and unstructured data (data held in scanned medical records, images, and transcriptions) into a single data platform for query. Automating this data flow is a focus of the Agency for Healthcare Research and Quality–funded SCOAP CERTN. Last, in its developmental years SCOAP has been lacking in collecting patient-reported outcomes (PROs), which may be the most important outcome as we hope to improve patient experience. This long-term PRO assessment is a key feature of spine, vascular, and urology modules, but is probably appropriate for all interventions (eg, post hernia pain). Creating the infrastructure for high-fidelity, long-term follow-up is resource intensive and is a novel component of SCOAP that may be difficult to replicate without considerable investment.

CONCLUSION

Five years after launch, SCOAP is thriving as it tries to create a “learning” healthcare system that improves care based on prior performance. SCOAP is now focusing on expansion into more healthcare settings (ambulatory centers and clinics), more clinical domains (urology, spine, and orthopedics), development of improved automated and linked data gathering, collection of PROs, and moving toward more “real-time” data feedback and interventions. Peer-based collaboratives require significant clinical leadership and harnessing of the professionalism and fellowship of the surgical community. The state chapter system of the ACS has been essential in driving SCOAP forward and may be a good model for others considering such a program. It is our experience that regional initiatives like SCOAP offer a flexible, adaptive, and locally sensitive complement to national programs. These regional initiatives may augment the report card orientation of national programs by engaging communities with more actionable data. The challenges to these programs are significant, but the opportunities for both QI and culture change cannot be overstated. Alternatives to programs like SCOAP are top-down and driven by stakeholders that often fail to recognize the nuances of clinical care and the role clinicians can play in really improving the quality of care when quality is measured correctly and for the right reasons. We look forward to updating the surgical community of SCOAP’s progress at regular intervals.

Acknowledgments

SCOAP is supported by a grant from Washington State’s Life Science Discovery Fund and Agency for Healthcare Research and Quality Grant Number 1 R01 HS 20025-01.

SCOAP is a program of the Foundation for Healthcare Quality (www.qualityhealth.org) whose leadership, including Terry Rogers MD and Rosa Johnson ARNP MN CPHQ, have tirelessly worked to promote, support, and maintain SCOAP. The University of Washington Department of Surgery provided key support during the development of SCOAP and remains a driving force in SCOAP’s success. The authors are indebted to Drs. Carlos Pellegrini (Chairman, Department of Surgery) and E. Patchen Dellinger (Chief, Division of General Surgery) for their support of SCOAP.

References

  • 1.Kohn KT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: National Academy Press; 1999. [PubMed] [Google Scholar]
  • 2.Wennberg JE. Time to tackle unwarranted variations in practice. BMJ. 2011;342:d1513. doi: 10.1136/bmj.d1513. [DOI] [PubMed] [Google Scholar]
  • 3.Merkow RP, Bilimoria KY, Cohen ME, Richards K, Ko CY, Hall BL. Variability in reoperation rates at 182 hospitals: a potential target for quality improvement. 2009;209:557–64. doi: 10.1016/j.jamcollsurg.2009.07.003. [DOI] [PubMed] [Google Scholar]
  • 4.Kao LS, Thomas EJ. Navigating towards improved surgical safety using aviation-based strategies. J Surg Res. 2008;145:327–35. doi: 10.1016/j.jss.2007.02.020. [DOI] [PubMed] [Google Scholar]
  • 5.MedQIC. SCIP project information. 2010 (cited 2010 Nov 10). Available from: https://www.qualitynet.org.
  • 6.Stulberg JJ, Delaney CP, Neuhauser DV, Aron DC, Fu P, Koroukian SM. Adherence to surgical care improvement project measures and the association with postoperative infections. JAMA. 2010;303:2479–85. doi: 10.1001/jama.2010.841. [DOI] [PubMed] [Google Scholar]
  • 7.Cossman DV. The taking. General Surgery News. 2010 Oct; cited 2010 Dec 12. Available from: http://www.generalsurgerynews.com.
  • 8.Omoigui NA, Miller DP, Brown KJ, et al. Outmigration for coronary bypass surgery in an era of public dissemination of clinical outcomes. Circulation. 1996;93:27–33. doi: 10.1161/01.cir.93.1.27. [DOI] [PubMed] [Google Scholar]
  • 9.Birkmeyer JD, Shahian DM, Dimick JB, et al. Blueprint for a new American College of Surgeons: National Surgical Quality Improvement Program. J Am Coll Surg. 2008;207:777–82. doi: 10.1016/j.jamcollsurg.2008.07.018. [DOI] [PubMed] [Google Scholar]
  • 10.Flum DR, Fisher N, Thompson J, Marcus-Smith M, Florence M, Pellegrini CA. Washington State’s approach to variability in surgical processes/Outcomes: Surgical Clinical Outcomes Assessment Program (SCOAP) Surgery. 2005;138:821–8. doi: 10.1016/j.surg.2005.07.026. [DOI] [PubMed] [Google Scholar]
  • 11.Birkmeyer NJ, Share D, Campbell DA, Jr, Prager RL, Moscucci M, Birkmeyer JD. Partnering with payers to improve surgical quality: the Michigan plan. Surgery. 2005;138:815–20. doi: 10.1016/j.surg.2005.06.037. [DOI] [PubMed] [Google Scholar]
  • 12.Malenka DJ, O’Connor GT. The Northern New England Cardiovascular Disease Study Group: a regional collaborative effort for continuous quality improvement in cardiovascular disease. Jt Comm J Qual Improv. 1998;24:594–600. doi: 10.1016/s1070-3241(16)30408-4. [DOI] [PubMed] [Google Scholar]
  • 13.Maynard C, Goss JR, Malenka DJ, et al. Adjusting for patient differences in predicting hospital mortality for percutaneous coronary interventions in the Clinical Outcomes Assessment Program. Am Heart J. 2003;145:658–64. doi: 10.1067/mhj.2003.182. [DOI] [PubMed] [Google Scholar]
  • 14.Mahid SS, Polk HC, Jr, Lewis JN, Turina M. Opportunities for improved performances in surgical specialty practice. Ann Surg. 2008;247:380–8. doi: 10.1097/SLA.0b013e31815efd7a. [DOI] [PubMed] [Google Scholar]
  • 15.Shively EH, Heine MJ, Schell RH, Sharpe JN, Garrison RN, Vallance SR, et al. Practicing surgeons lead in quality care, safety, and cost control. Ann Surg. 2004;239:752–60. doi: 10.1097/01.sla.0000128301.67780.d7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Cuschieri J, Florence M, Flum DR, et al. Negative appendectomy and imaging accuracy in the Washington State Surgical Care and Outcomes Assessment Program. Ann Surg. 2008;248:557–63. doi: 10.1097/SLA.0b013e318187aeca. [DOI] [PubMed] [Google Scholar]

RESOURCES