Objective Assessment of Quality Measurement and Improvement

Abstract

Accurate quality measurement that allows for and results in improvement is essential to colon and rectal surgery. Currently, no consensus exists as to which variables are most important in measuring outcomes. Debate continues concerning the “best” variables to measure from a structural, process, and outcomes standpoint. Although American College of Surgeons National Surgical Quality Improvement Program provides an opportunity for outcomes measurement in general and vascular surgery, there is no specific quality improvement tool available for colon and rectal surgery. However, there is growing literature testing the validity of candidate variables to be used in such a data collection system. This article evaluates the current objective assessment measurements used for quality improvement in colon and rectal surgery.

CME Objective: On completion of this article, the reader should be able to summarize the variables currently being measured for quality improvement in colon and rectal surgery.

Quality measurement in colon and rectal surgery is essential to improve outcomes. However, the identification of the key variables to measure is difficult. Currently, there is a lack of consensus on defining the most appropriate variables to efficiently monitor and improve the quality of care. In fact, the question of which variables to measure is one of considerable debate.

The most prevalent quality improvement programs in use today for colon and rectal surgery were developed for general surgery and thus may not be specific enough to provide the best measure of quality in colon and rectal surgery. They include the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP), the Surgical Care Improvement Project (SCIP), and the Surgical Care Outcomes Assessment Program (SCOAP). Details of these programs will be discussed later.

Quality Measures

Quality measures can be broadly divided into those concerning structure, process, and outcomes.¹ Each of these categories is not without limits, yet they do provide a basic blueprint by which quality can be measured. The authors have used all three as tools to define and improve quality. Examples and a comparison of variable types can be found in Table 1.

Table 1. Overview of quality improvement variables¹ .

	Structural	Process	Outcomes
Examples	Colectomy volume Colorectal fellowship training Hospital volume	Ostomy nurse consulted preoperatively for low anterior resection (LAR) or planned ostomy Antibiotic given 1 h before incision	Morbidity and mortality Cost Re-admission rates
Pros	Easy and inexpensive to measure Use for certification	Measures exactly what patients receive Actionable items	Measures and results
Cons	Produces few actionable items Imperfect proxies for outcomes Uncertain quality	Best process variables are not generalizable (too specific to procedure) Generalized processes may not be effective for specific procedures	Sample size too small for most institutions Outcomes may not be procedure specific

Structural

Structural variables describe physical attributes, organizational structure, or staff qualities. These variables, such as procedure volume, are fairly easy to measure and can often be done so with hospital administrative data.¹

Billingsley et al examined the structural variable of volume and its impact on outcomes in colon cancer surgery.² Patients of high-volume surgeons (17–26 operations per year) were found to have lower postoperative mortality rates. Patients treated by very high-volume surgeons (27–85 operations per year) had a lower rate of postprocedural intervention for surgical complications in addition to a lower postoperative mortality rate. Hospitals categorized as very high volume (151–341 operations per year) were independently associated with lower postoperative patient mortality rates. An additional surgeon factor that may provide further insight, and acts as a surrogate for quality, is fellowship training. Porter et al reported that patients with rectal cancer who were resected by surgeons with formal colon and rectal fellowship training experienced a decreased 5-year local recurrence rate and an increased disease-specific survival.³

Hospital classification or designation is another structural variable representing a crude measure of quality. Paulson et al demonstrated that a National Cancer Institute (NCI) designation was associated with a lower risk of postoperative mortality and improved long-term survival after colectomy for colon and rectal cancer, independent of volume.⁴ However, the association between NCI designation and quality may be confounded by the presence of specialists and/or high-volume centers or practices. Other previously studied structural variables include hospital participation in clinical trials, intensivist staffing of intensive care units, and high nurse-to-bed ratios.⁵

One limitation of using structural variables for quality measurement is the lack of actionable items for improvement and the uncertain proxy for quality that these variables represent.¹ Although a hospital can measure outcomes by surgeon volume, percentage of board certified surgeons, or NCI status, it is difficult to implement measures to change outcomes. Furthermore, most structural variables measure associations, not causation. In fact, a recent study comparing outcomes for six cancer types at both NCI and non-NCI designated hospitals demonstrated no improvement in long-term survival rates.⁵ Although structural variables provide a window into potential measures that improve quality, they are imperfect and not directly actionable.

Process

Process variables refer to the care received by patients.¹ A common example is discharging patients on a β-blocker following a myocardial infarction. Process variables provide the advantage of being directly actionable for providers. An example of process measurement in surgery is the SCIP.

SCIP was founded by a partnership of multiple organizations in 2002. Originating as the National Surgical Infection Program, SCIP was created with the goal of reducing surgical complications by 25% by 2010.^{6 7} It targeted the most common postoperative complications: surgical site infection (SSI), venous thromboembolism, and cardiac events. Infection prevention process measures were based on evidence as well as expert opinion. These measures included antibiotic administration within 1 hour before incision, administration of the proper antibiotic, discontinuation of prophylactic antibiotics within 24 hours of surgery completion, compliance with 6 am postoperative blood glucose control (cardiac surgery), rate of postoperative wound infection diagnosed during hospitalization, appropriate hair removal before surgery, and postoperative normothermia (colorectal surgery). The current SCIP infection prevention processes can be found in Table 2.⁸

Table 2. SCIP infection process measures⁸ .

SCIP Inf-1: Antibiotic administration within 1 h before incision SCIP Inf-2: Administration of the appropriate antibiotic SCIP Inf-3: Discontinuation of prophylactic antibiotics within 24 h of surgery end SCIP Inf-4: Compliance of 6 am postoperative blood glucose control in (cardiac surgery) SCIP Inf-6: Appropriate hair removal before surgery SCIP Inf-9: Urinary catheter removed on postoperative day (POD) 1 or POD 2 SCIP Inf-10: Maintaining normothermia after colorectal surgery

Abbreviation: SCIP, Surgical Care Improvement Project.

However, adherence to SCIP process measures has not universally translated into improved outcomes. A cross-sectional study of 200 hospitals examined the correlation between compliance with four SCIP SSI-related process measures and outcomes.⁹ These outcomes were defined based on ACS NSQIP definitions (see Table 3) and grouped into overall morbidity, serious morbidity, mortality, and SSI.¹⁰ With an overall hospital SCIP compliance of 95%, 15 of the 16 correlations were found be nonsignificant. The only significant correlation was the administration of the appropriate prophylactic antibiotic with SSI.

Table 3. ACS NSQIP morbidity and mortality variables¹⁰ .

ACS NSQIP reported outcomes  • SSI (superficial, deep, or organ space)  • Wound disruption  • Pneumonia  • Unplanned intubation  • Pulmonary embolism  • Failure to wean from ventilator for > 48 h  • Progressive renal insufficiency  • Urinary tract infection  • Cerebrovascular accident  • Coma lasting > 24 h  • Peripheral nerve injury  • Cardiac arrest  • Myocardial infarction  • Bleeding requiring > 4 units of packed red blood cells or whole blood within 72 h postoperatively  • Graft/prosthesis/flap failure  • Deep venous thrombosis/thrombophlebitis  • Sepsis  • Septic shock  • Death

Abbreviations: ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Program; SSI, surgical site infection.

The association between SCIP compliance and outcomes has also been studied in colon and rectal surgery. A 2010 study by Pastor et al created a multidisciplinary task force to increase compliance with SCIP measures, and then evaluated the outcomes.¹¹ Over the course of the study period, there was an increase in all SCIP infection prevention processes, except for control of perioperative glucose levels. Despite the increase in compliance, the SSI rates did not decrease in patients undergoing segmental resections.¹¹ These findings should not be interpreted as an inability of process measures to improve quality. In fact, a recent study demonstrated the use of a Comprehensive Unit-Based Safety Program (CUSP) to reduce SSIs.¹² Among the processes evaluated and standardized were skin preparation for colon and rectal surgery, elimination of routine pre-operative mechanical bowel preparation, and enhancement of sterile techniques. In the year after CUSP, SSI rates dropped from 27.3 to 18.2%. The Better Colectomy Project evaluated 37 evidence-based best practices from surgical literature and expert opinion. Of these processes, 15 were determined to be “key processes” for colectomy.¹³ They were categorized as infection prevention, thromboembolism prophylaxis, and preoperative assessment and optimization. Adherence rates with all best practices and key practices were measured, as was their association with 30-day morbidity and mortality as defined by ACS NSQIP. The study demonstrated that nonadherence to key processes predicted the occurrence of a complication, and each additional missed process increased the odds of a postoperative complication by 60%.

When matched appropriately to a specific procedure, process measures can have a significant impact. However, if process measures are too broad, increased compliance will not translate into better outcomes. Currently, there is limited evidence studying procedure-specific process measures and improved quality. Much of the existing data relies on expert opinion.

Outcomes

Outcomes measurement provides a “bottom-line” end result in quality improvement.¹ Outcomes may include length of stay (LOS), number of nodes harvested in a colectomy for cancer, reoperation, morbidity, mortality, and cost.

The most prevalent quality improvement system using outcomes measurement is the ACS NSQIP.

The original NSQIP was developed in the 1990s after Veteran's Administration (VA) hospitals came under scrutiny for the perceived low quality of care in their hospitals.¹⁴ The VA was tasked with comparing its surgical morbidity and mortality rates against the national average. In response, the VA conducted the National Surgical Risk Study to identify risk adjusted preoperative predictors of morbidity and mortality.^{15 16} Expected morbidity and mortality rates were compared with the rates observed in hospitals included in the study. Processes of care at hospitals with low observed-to-expected (O/E) ratios were found to be superior to those at hospitals with high O/E ratios.

After the launch of NSQIP, the VA saw a staggering 47% drop in postoperative mortality and a 43% drop in morbidity from 1991 to 2006.¹⁶ In 1999, a pilot study in general and vascular surgery determined that the NSQIP model could effectively be used in non-VA environments as well.¹⁴ A 2002 Institute of Medicine consensus report cited NSQIP as a major factor in naming the VA health care system the “Best in the Nation.”¹⁷ In 2004, the ACS began enrolling additional hospitals into what is now known as ACS NSQIP. This registry uses clinical data from medical records collected by a trained data extracter.¹⁸ It focuses on 30-day outcomes of 21 specifically defined morbidities and mortality.¹⁰ Currently, more than 500 hospitals in the United States participate in ACS NSQIP. It has become the first nationally validated, risk-adjusted, outcomes-based program to measure and improve the quality of surgical care across surgical specialties in the private sector. Hall et al demonstrated that participation in NSQIP improves outcomes across all hospital types in the private sector independent of the initial O/E ratio.¹⁸

Several authors have investigated outcomes in colon and rectal surgery using ACS NSQIP as the primary data source. McCloskey et al demonstrated that laparoscopic colectomy in high-risk patients is safe as evidenced by a lower morbidity and mortality rate compared with the open group.¹⁹ However, this study, like many others targeting a specific outcome, was grossly underpowered. Furthermore, surrogates for improved quality, which would be specific to colon and rectal surgery, such as anastomotic leak, are not currently captured by ACS NSQIP, impeding efforts at disease-specific and specialty-specific quality improvement. In response to this, ACS NSQIP has developed a “procedure-targeted” program of 30 higher risk procedures that began to roll out in 2011.²⁰ Targeted procedures require a minimum of 1,680 cases to be collected annually.²¹ Under this program, hospitals may collect additional procedure-specific data and measure procedure-specific outcomes, in addition to the standard outcomes captured by ACS NSQIP.

Hybrid

Not all quality measurement systems fall neatly into one of the previously discussed categories. The SCOAP is a physician-led collaborative of hospitals and surgeons in the State of Washington whose primary goal is to improve quality by reducing variation in outcomes and process of care at every hospital in the region.²² A recent study by Kwon et al using data collected from 40 participating hospitals in the SCOAP collaborative evaluated the use of routine anastomotic leak testing during left-sided colon or rectal resections.²³ The study demonstrated that hospitals performing routine leak testing had a reduction of greater than 75% of composite adverse events. The authors further suggested adding routine leak testing for left-sided colon or rectal resections as a process-of-care metric to improve quality. The major limitation to SCOAP is its lack of generalizability. SCOAP is funded by the state and limits participation of hospitals from other states. However, SCOAP does provide a blueprint for the implementation of a quality measurement and improvement system.

Improving Quality Measurement in Colorectal Surgery

Quality measurement has improved over the past several decades yet many obstacles remain. First, ACS NSQIP, the primary data source for surgical quality improvement, and similar databases were not developed specifically for colon and rectal surgery. Process and outcome measures, which may be unique to colon and rectal surgery, are not included creating a virtual “blind spot” in quality measurement and assessment.²⁴ Second, there is a lack of consensus on the best outcomes to measure. McGory et al sought to develop and rate quality indicators for patients undergoing colon and rectal cancer surgery using published literature and expert opinion.²⁵ Initially, 142 candidate indicators were identified. After a literature-based validation process and further expert opinion, 92 indicators remained. These ranged from pre- to postoperative care for colon and rectal cancers. Similarly, the American Board of Colon and Rectal Surgery (ABCRS) and the American Society of Colon and Rectal Surgeons (ASCRS) sought candidate end points for the use in developing surgeon-specific registries for case reporting and quality improvement.²⁶ Using a modified Delphi method, a list of structural, process, and outcomes measures were created and graded in successive rounds to evaluate their importance. However, no low-scoring metrics were removed, as there was no objective cutoff point. Upon conclusion, 89 quality measurements were created. The goal of this study was not to create a concise list of variables, but rather to create a starting point toward generation of standardized end points. Interestingly, the highest-ranking quality measurement was anastomotic leak, which is not captured by ACS NSQIP, SCIP, or SCOAP. Third, there is a significant burden on hospitals and physicians for increased data collection and reporting. ACS NSQIP data are collected by a trained surgical clinical reviewer. A large expansion of metrics to be documented and collected may place the burden on physicians, adding to their increasing nonclinical responsibilities.²⁶ Furthermore, data collected in a nonsystematic fashion could lead to imperfect reporting. Finally, government and payer-directed quality improvement programs may be viewed as punitive, creating a defensive mentality among clinicians.²² This could lead to the underreporting of adverse outcomes.

The ideal quality measurements would be universally agreed upon, specific to each procedure, easy and free to collect, and serve as a perfect proxy for outcomes. However, Cook and Hyman state that pragmatically, a colon and rectal quality improvement system should seek usefulness, not perfection. This ideal encompasses a balanced set of structural, process, and outcomes variables. The system should keep measurement simple, using both qualitative and quantitative data. Finally, it must strictly define operational measures and incorporate data collection into the daily work routine.²⁷

Efforts to create a quality measurement and improvement system for colon and rectal surgery have already begun. As NSQIP was being developed by the VA for general surgery, to determine expected outcomes for procedures, the Portsmouth Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (P-POSSUM) was created in the United Kingdom.²⁸ This system has been applied to colon and rectal surgery, though numerous studies have enumerated its limitations. In response to this, a colon and rectal risk-adjusted scoring system was developed (CR-POSSUM). A study of 15 National Health Services hospitals in the United Kingdom demonstrated this specialty-specific model to be an accurate predictor of colon and rectal operative mortality.²⁹ In 2009, ACS NSQIP similarly published a colon and rectal surgery risk calculator and measured it against their previously defined outcomes.³⁰ The colon and rectal risk calculator was able to predict outcomes just as well, or better than those reported in the literature. With more specific risk calculators, evaluating hospitals with low O/E postoperative complication rates may help drive the development of more effective structural, process, or outcomes measures.

The Best Variables to Measure Quality and Track Improvement

As mentioned earlier, members of ASCRS and ABCRS attempted to build consensus-based quality end points for colon and rectal surgery.²⁶ As the subspecialty covers a broad range of pathologies, the first goal was to group outcomes into disease and procedural-based categories. These categories were colectomy, rectal cancer, hemorrhoidectomy, anal fistula and abscess, colonoscopy, and rectal prolapse. Proposed quality metrics were created using the modified Delphi method to create a consensus derived ranked list of 89 processes and outcome measures. A list of the highest scoring factors from each category can be seen in Table 4. Categorizing outcomes allows measurements to be very specific to each disease or procedure type. These 89 variables should serve as a starting point in developing a colon and rectal data registry tool to improve quality. However, many of the listed end points may be interpreted as “important considerations” rather than specific process or outcomes variables to be measured. This finding suggests the need to refine these considerations into measurable outcomes that can be tracked and used for improvement. Also, as many of the proposed quality measurements were based solely on expert opinion, they have yet to be validated by the literature.

Table 4. Top three quality metrics per category²⁶ .

Colectomy  1. Anastomotic leak  2. Indication for surgery  3. TNM stage Rectal cancer  1. Neoadjuvant therapy given  2. Circumferential margin  3. Anastomotic leak Hemorrhoidectomy  1. Operation  2. Reoperation  3. Sphincter injury with incontinence Anal fistula/abscess  1. Fistula repair type  2. Presence of inflammatory bowel disease (IBD)  3. Procedure for recurrence Colonoscopy  1. Perforation  2. Polypectomy  3. Appropriate Indication Rectal prolapse  1. Approach  2. Resection performed (Y/N)  3. Fixation (mesh/suture/none)

In 2002, a multidisciplinary panel from the Surgical Oncology Program of Cancer Care Ontario was created for the similar purpose of creating a colon and rectal quality improvement system.³¹ However, this panel focused specifically on colon and rectal cancers. Using a modified Delphi process, the panel created 42 quality care indicators that were reduced to 15 through successive rounds of panels and literature review. These indicators included the following areas: cancer screening detection method, preoperative evaluation, consultation with a medical or radiation oncologist, operative report information, lymph nodes harvested, margin status, anastomotic leak, mortality, local recurrence, survival rate, and surveillance colonoscopy. Vergara-Fernandez et al evaluated whether these 15 quality care indicators could be used to compare a single institution's colon and rectal cancer outcomes to standards published in the literature, to measure their own quality.³² Data were collected over a 10-year span on 13 of the 15 indicators. The study found that the institution's rate of postoperative mortality, 5-year survival, anastomotic leak, resection margins, lymph node harvest, and receipt of appropriate postoperative chemotherapy/radiation were all equivalent to, or better than those published in the literature. This study demonstrates how a single institution can use well-defined consensus derived quality measures to track their own quality. It remains to be seen if consensus derived measures, such as the one proposed by the Surgical Oncology Program of Cancer Care Ontario, can be applied broadly to institutions nationwide.

Although some authors have sought to define very specific quality metrics, others have chosen more broad measures for study. Morris et al examined reoperation as a broad quality indicator in colon and rectal surgery.³³ Data from patients with stages I to III colon and rectal cancers from the SEER-Medicare database were analyzed for the end point of any postoperative procedural intervention. Indications for intervention included (from most to least common) abdominal infection/abscess, wound infection/separation, organ injury/laceration, unspecified laparotomy, shock/hemorrhage, and retained foreign body. By using the metric of reoperation, the study measured all morbidities that required intervention as a group. However, this variable fails to capture morbidities that do not require procedural intervention, such as a nonoperatively managed anastomotic leak or a deep venous thrombosis.

A generic metric that may capture some of the factors neglected by postprocedural intervention is LOS. Cohen et al evaluated LOS for patients in 182 hospitals following a broad range of colon and rectal surgeries.³⁴ Patients were grouped by the presence or absence of postoperative complications. Extended LOS was defined as a hospital stay greater than the 75th percentile for the group. In patients without complications, 15 of the 182 hospitals were found to have a greater than expected number of patients with extended LOS, whereas 15 hospitals were found to have fewer than the expected number of patients with extended LOS. In the 175 hospitals that reported having at least one patient with a complication, only one had a greater than expected number of patients with extended LOS. However, this variable has several limitations. First, there is no direct causal relationship between procedure type and LOS. This is due to the possibility of confounding variables, such as the presence of comorbidities, complexity of surgery, complications, etc. Second, LOS alone as a measure of quality does not capture minor complications that do not translate into increased LOS, such as urinary tract infections.

Finally, a variable that may capture both postprocedure intervention and LOS is procedure-specific cost. With hospital and physician reimbursement set to decrease significantly over the next decade, hospitals will need to lower costs while maintaining the high standards of quality to remain competitive. Jensen et al evaluated the cost effectiveness of laparoscopy versus open resection for colon and rectal cancers.³⁵ Although laparoscopy was associated with increased operating room costs, it was also associated with decreased LOS, a lower rate of postoperative wound infection and ileus. These decreases amounted to an overall cost savings of $4,283 per laparoscopic procedure while generating quality of life outcomes equivalent to open surgery.

Summary

Quality measurement and improvement has changed significantly in colon and rectal surgery over the past two decades. Current quality measures may be divided into structural, process, or outcomes variables. Structural measures include demographic information, such as surgeon and hospital volume, fellowship training status, and hospital designation. Process measures include actions that can be done directly for the patient, such as giving antibiotics 1 hour before surgery. Outcomes variables are end results that can be measured, such as SSI, deep venous thrombosis, or death. Each type of variable has advantages and limitations.

Although the ideal quality measure would be specific to a colon and rectal procedure, be a perfect proxy for outcomes, and be easy and free to collect, no such measure currently exists. Attempts to define appropriate measurement tools have yet to be fully validated. A list of key publications on this subject can be found in Table 5.

Table 5. Key publications.

	References
Quality Overview	1
ACS NSQIP	14 18
SCIP	9
SCOAP	22
Development of Colorectal Outcome Measures	26

Abbreviations: ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Program; SCIP, Surgical Care Improvement Project; SCOAP, Surgical Care Outcomes Assessment Program.

The future of quality measurement and improvement in colon and rectal surgery will occur in the rapidly changing landscape of modern medicine. It will have to adapt to emerging technologies, new surgical techniques, and account for limited financial resources, along with an aging patient population. Although this remains a daunting task, its success is integral in the sustainability and viability of our current health care system.