Development and Assessment of Memorial Sloan Kettering Cancer Center’s Surgical Secondary Events Grading System

Abstract

Background

Studying surgical secondary events is an evolving effort with no current established system for database design, standard reporting, or definitions. Using the Clavien-Dindo classification as a guide, in 2001 we developed a Surgical Secondary Events database based on grade of event and required intervention to begin prospectively recording and analyzing all surgical secondary events (SSE).

Study Design

Events are prospectively entered into the database by attending surgeons, house staff, and research staff. In 2008 we performed a blinded external audit of 1,498 operations that were randomly selected to examine the quality and reliability of the data.

Results

1,498 of 4,284 operations during the 3^rd quarter of 2008 were audited. 79% (N=1,180) of the operations did not have a secondary event while 21% (N=318) of operations had an identified event. 91% (1,365) of operations were correctly entered into the SSE database. 97% (129/133) of missed secondary events were Grades I and II. Three Grade III (2%) and one Grade IV (1%) secondary event were missed. There were no missed Grade 5 secondary events.

Conclusion

Grade III – IV events are more accurately collected than Grade I – II events. Robust and accurate secondary events data can be collected by clinicians and research staff and these data can safely be used for quality improvement projects and research.

Introduction

Surgical quality improvement requires the accurate and reproducible categorization of surgical secondary events (SSEs) within a validated system¹. In 1992, Clavien² proposed the a standardized system, modified in 2004 by Dindo³, based on the intervention needs and health status of the patient. While other standards have been proposed^4-7, the Clavien-Dindo classification⁸ remains the most common method for grading SSEs.

Developed within the Veterans Affairs Health System⁹, the American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP) has since expanded¹⁰ and is the only specialty-wide national quality improvement project of its kind. NSQIP has shown that clinical data are superior to administrative data in identifying SSEs^11,12 , and has lead to a decrease in SSEs at participating hospitals¹³. NSQIP has been widely adopted and provides bench-marking of events between hospitals, and the NSQIP Surgical Risk Calculator now leverages NSQIP data to provide patient-specific estimates of post-operative outcomes¹⁴. However, NSQIP does not capture 100% of the operations at participating institutions, its records are not warehoused with other clinical information (financial, clinical, or pathologic), and there are important SSEs (including anastomotic leak and pancreatic fistula) that are not captured by NSQIP, and NSQIP does not record the severity of any events.

In 2001, the Department of Surgery at Memorial Sloan-Kettering Cancer Center (MSKCC) developed the institutional Surgical Secondary Events (SSE) Database to track deviations from the expected post-operative course that occur within the first 30 post-operative days. We modified the Clavien-Dindo classification system and further defined specific secondary events by body system as listed in Table 1 Currently, the SSE database classifies over 220 distinct surgical secondary events, in 14 physiologic categories, designed for a department whose procedures are almost entirely limited to the treatment of cancer. Since all surgical disciplines are in the Department of Surgery, the secondary events classification system includes all of the surgical subspecialties, which accounts for the large number of possible events. Every operation performed is reviewed and included in the database, which in 2008 contained over 80,000 distinct operations and currently includes over 160,000.

Table 1.

Memorial Sloan Kettering Cancer Surgical Secondary Events Database Classifications. The SSE database classifies all deviations from the expected post-operative course with a modification of the Clavien-Dindo classification.

Grade	Surgical Secondary Event Requiring or Resulting in:
Grade 1	Bedside Care or Oral Medications
Grade 2	Intervenous Medications, Transfusion
Grade 3	Radiologic, Endoscopic, or Operative intervention required
Grade 4	Chronic Disability or Organ Resection
Grade 5	Death
Body Systems
Cardiovascular System	Infection
Endocrine System	Metabolic
Gastrointestinal System	Musculoskeletal System
General	Nervous System
Genitourinary System	Pain
Head and Neck	Pulmonary System
Hematologic or Vascular System	Wound or Skin

One of the critical goals of this system is to ensure complete and accurate collection of all SSEs. The quality of the data collection determines the accuracy of the SSE database, which is critical in ensuring accurate SSE analysis and quality improvement projects. To evaluate the quality of our data, in 2009 we conducted a blinded external audit of the SSE database.

Methods

Data Entry

The secondary events database is maintained within our electronic medical record (EMR) and contains a list of all surgical procedures, and any associated SSEs, the patient has undergone since its establishment. In cases where a patient experienced multiple SSEs they are listed chronologically with their associated operation. Data for every operation are captured in a prospective, institution-wide, three-tiered approach involving attending surgeons, house staff, and research staff. Data is gathered at the point of care, on chart review, during morbidity and mortality conferences, and at patient follow up visits. Every event and corresponding grade has a strict definition, and events are entered in real time by the surgical staff. To record an SSE the physician selects the operation and enters the event, its grade, and the date the SSE occurred.

Though most events are recorded as they occur, an additional layer of surveillance occurs at Morbidity and Mortality conferences when each attending reviews the SSE record of patients who are more than 30 days post-op, add any unrecorded events occurring within the first 30 days and corrects any entries. Data are reconciled and updated in the system, resulting in 100% capture and reporting. This additional layer or surveillance requires 25% effort of a research staff member for each surgical service. Readmissions are not graded as a specific grade, rather based on the required care they receive while hospitalized. A full list of the categories, specific secondary events, and corresponding grades, is available for download at www.mskcc.org/sse.

Audit

This audit was approved by MSKCC’s Institutional Review Board and was conducted in 2009. . Using an expected discrepancy rate between clinical documentation and the SSE database of 15%, the 95% confidence interval (CI) for sampling various numbers of operations plateaued at 1,500 operations with a 95% CI of 13.2% to 16.9%, while a sample size of 2,500 operations would have resulted in a 95% CI of 13.6% to 16.5%. A random sample of 1,500 operations, from the 4,284 performed during the third quarter of 2008, were selected for audit. . The audit focused on the review of all surgical secondary events to assess whether SSEs were accurately recorded in the database and whether the recorded SSEs were supported by the clinical documentation in the EMR.

National Audit, a Florida based medical billing audit company performed the audit. National Audit nurses were assigned individual patients and given full EMR access. They first reviewed the clinical documentation to identify all secondary events and then compared their recorded events to the entries in the SSE database. Any discrepancies between then EMR and the SSE database were adjudicated by two attending surgeons who were not associated with the primary case.

At the same time as the chart review, a 150 patient subset was selected for a focused telephone audit to assess whether patient required care or hospitalization at another institution subsequent to their discharge from MSKCC. Research staff asked a series of scripted questions about their initial hospital and post-operative course in an attempt to identify previously unknown secondary events that resulted in care at other institutions.

Results

1,498 operations were included in the final audit from the 4,284 surgeries performed during the 3^rd quarter 2008. Most operations (79%; 1,180 / 1,498) had no SSE while 21% (N=318) had an identified event. Overall, 7% (N=110) had a Grade 1 event, 9% (N=139) had a Grade 2 event, 4% (N=60) had a Grade 3 event, 0.27% (N=4) had a Grade 4 event and 0.33% (N=5) had a Grade 5 event (Table 2). Overall, 91% of operations were correctly identified in the SSE database. The overall SSE event rate by each service, de-identified for this report, is listed in Table 2, Table 3 lists the five most common events per grade.

Table 2.

Service rates of Surgical Secondary Events. Services were de-identified for the audit but included Breast, Colorectal, Gastric and Mixed Tumor, Gynecology, Head and Neck, Hepatopancreaticobiliary, Orthopedics, Pediatric Surgery, Plastic and Reconstructive Surgery, Thoracic Surgery, and Urology.

Service	Operations Audited	Operations with Events	% Operations with Events	Total Grade 1	% of Service Total	Total Grade 2	% of Service Total	Total Grade 3	% of Service Total	Total Grade 4	% of Service Total	Total Grade 5	% of Service Total
Service A	308	30	10%	20	6%	8	3%	2	1%	0	0%	0	0%
Service B	88	33	38%	10	11%	16	18%	6	7%	1	1%	0	0%
Service C	95	21	22%	6	6%	9	9%	5	5%	0	0%	1	1%
Service D	116	41	35%	14	12%	26	22%	1	1%	0	0%	0	0%
Service E	131	16	12%	10	8%	3	2%	2	2%	1	1%	0	0%
Service F	62	23	37%	5	8%	9	15%	9	15%	0	0%	0	0%
Service G	56	13	23%	2	4%	4	7%	6	11%	0	0%	1	2%
Service H	43	14	33%	5	12%	4	9%	4	9%	1	2%	0	0%
Service I	93	10	11%	4	4%	5	5%	1	1%	0	0%	0	0%
Service J	213	56	26%	15	7%	33	15%	6	3%	0	0%	2	1%
Service K	293	61	21%	20	7%	22	8%	17	6%	1	0%	1	0%
Department Totals	1498	318	21%	111	7%	139	9%	59	4%	4	0.27%	5	0.33%

Table 3.

Five most common Surgical Secondary Events per grade.

Grade 1			Grade 2
Name	#		Name	#
Wound Complication *		41	Cardiac Arrhythmia **		22
Cellulitis		14	Anemia		16
Urinary Tract Infection		10	Cellulitis		16
Pleural Effusion		7	Wound Infection		9
Anemia		5	Urinary Tract Infection		8
Grade 3			Grade 4
Name	#		Name	#
Anastomotic Leak ***		8	Facial Nerve Palsy		1
Intra-abdominal Infection or Abscess		5	Liver Dysfunction / Failure		1
Hematoma		4	Renal Failure		1
Hemorrhage		4	Vascular Thrombosis		1
Deep Venous Thrombosis		3
Grade 5
Name	#
Progression of disease, death		2
Aspiration		1
Dehydration		1
Sepsis		1

^*19 wound infections, 13 wound breakdowns, 9 seromas

^**20 supraventricular arrhythmias, 2 ventricular arrhythmias

^***Anastomotic Leak = 2 Pancreatic, 2 Urinary, 1 Biliary, 1 Intestinal, 1 Pharyngeal, 1 Rectal

Of the 21% (N=318) of operations with an SSE identified, 58.3% (N=186) matched between the EMR and the database, while 41.7% (N=133) of SSEs were not recorded in the database (Table 4). Of the unrecorded events, 97% (129) were either Grade 1 (74, 56%) or Grade 2 (55, 41%). Events of these grades require either bedside care and PO medications (Grade 1) or IV medications or transfusions (Grade 2) as treatment (Table 1). Importantly, less than 3% (N = 4; 3 Grade 3; 1 Grade 4) of Grade 3 or 4 events, those requiring radiologic or surgical intervention, intubation, or resulting in chronic disability, were not recorded. All Grade 5 secondary events were recorded. Table 4 lists the correctly reported and missed events by de-identified surgical service.

Table 4.

Breakdown of Surgical Secondary Events discovered in the EMR not reflected in the SSE Database. Services were de-identified for the audit but included Breast, Colorectal, Gastric and Mixed Tumor, Gynecology, Head and Neck,Hepatopancreaticobiliary, Orthopedics, Pediatric Surgery, Plastic and Reconstructive Surgery, Thoracic Surgery, and Urology.

Service	Total Operations Audited	Total Correct	Total Missed	% Missed of Service Total	% Missed of Dept Total	Grade 1	% of Grade	Grade 2	% of Grade	Grade 3	% of Grade	Grade 4	% of Grade
Service A	308	287	21	7%	16%	15	20%	6	11%	0	0%	0	0%
Service B	88	77	11	13%	8%	3	4%	8	15%	0	0%	0	0%
Service C	95	86	9	9%	7%	4	5%	4	7%	1	33%	0	0%
Service D	116	111	5	4%	4%	3	4%	2	4%	0	0%	0	0%
Service E	131	119	12	9%	9%	9	12%	2	4%	0	0%	1	100%
Service F	62	55	7	11%	5%	3	4%	4	7%	0	0%	0	0%
Service G	56	53	3	5%	2%	1	1%	2	4%	0	0%	0	0%
Service H	43	39	4	9%	3%	3	4%	1	2%	0	0%	0	0%
Service I	93	87	6	6%	5%	4	5%	2	4%	0	0%	0	0%
Service J	213	187	26	12%	20%	13	18%	12	22%	1	33%	0	0%
Service K	293	264	29	10%	22%	16	22%	12	22%	1	33%	0	0%
Dept Totals	1498	1365	133		100%	74	100%	55	100%	3	100%	1	100%

Common Grade 1 and Grade 2 Secondary Events

Overall, 110 (7%) of the patients in our audit experienced a Grade 1 secondary event. Many of these events (41, 37%) were related to the patients surgical wound (19 wound infections, 13 wound breakdowns, and 9 seromas), and another 14 patients (12%) developed a cellulitis that required oral antibiotics.

Of the 9% of patients (139 / 1,498) who experienced a Grade 2 event, the most common was a cardiac arrhythmia. Another 16 patients developed anemia requiring a transfusion (5 patients developed a Grade 1 anemia), 16 patients developed cellulitis that required IV antibiotics, and 9 patients developed a wound infection that required IV antibiotics.

Common Grade 3 Events

Four of the five most common Grade 3 events (Anastomotic leak; Intra-abdominal Infection or Abscess; Hematoma; Hemorrhage) are directly related to the operative intervention, and the other (deep venous thrombosis, DVT) is a direct consequence of our patients’ hypercoagulability. We combined all the anastomotic leaks into one category although their individual incidence (2 Pancreatic, 2 Urinary, 1 Biliary, 1 Intestinal, 1 Pharyngeal, and 1 Rectal) was rare. None of these leaks resulted in long term disability.

Grade 4 Events

One patient within our audit suffered permanent facial nerve palsy as a direct result of resection of a malignancy involving the facial nerve, and possibility that was thoroughly discussed with the patient pre-operatively. Additionally, one patient each suffered permanent liver dysfunction, renal failure, and vascular thrombosis.

Grade 5 Events

There were five patients within our audit who suffered Grade 5 events. Two patients succumbed due to progression of their disease, while one patient each expired from aspiration, dehydration, and sepsis.

Unrecorded Events of all Grades

There was agreement between the EMR and the SSE database in 91% of the patients. There were 133 events that were not recorded in the SSE database (Table 4). Of these unrecorded events, 56% (74) were Grade 1 and 41% (55) were Grade 2, leaving only three (2%) unrecorded Grade 3 events and the one (1%) Grade 4 event, the facial nerve palsy previously discussed.

Events in the SSE Database not Documented in the EMR

There were 25 events in the SSE where supporting documentation could not be identified in the EMR. Classifying the EMR as the gold standard, the overall sensitivity and specificity of the database is 54% and 97%; for Grade 1 – 2 events it is 45% and 99%, and for Grade 3 – 5 events it is 93% and 99%.

Telephone Survey

We were able to contact 87 (58%) of the 150 patient subset randomly selected for telephone followup, 82 of whom agreed to participate (94% of patients contacted). The majority (68%, 56 / 82) reported they had no SSEs post-operatively, while 26 (32%) of the patients reported an event within 30 days. Of the 26 patients who self-reported an SSE in the first 30 days, 21 (81%) self-reported secondary events incorrectly. The discrepancy was due to either reporting an issue that did not, upon clinician review, meet the definition of a secondary event or incorrectly reporting an event that occurred during their hospitalization as occurring post-discharge. Only three patients (4%) reported seeking care at an outside facility in their initial post-operative period. Two were seen in their local emergency department (one within 30 days, and one within 60 days of their operation), and the third sought care with their primary care physician.

Discussion

The MSKCC SSE database accurately records post-operative secondary events occurring within the first 30 post-operative days. It was established in 2001 as an evolving, prospective database. By 2008 there were over 80,000 surgical procedures within the database, 15% of which had a secondary event. In order to assess the integrity of the data, in 2009 we underwent an outside audit of the database and examined procedures that took place during the third quarter of 2008. The results of our external audit demonstrate that our data collection is accurate.

Our database has previously been used to study our institutional experience with pancreaticoduodenectomy¹⁵. At that time, prior to our audit, the database accurately collected events on Grade 3 – 5 secondary events but did not capture all Grade 1 and 2 secondary events. Upon chart review the SSE database was correct in 81% of operations, while our review found a 91% concordance. We believe this improvement is a consequence of the continued emphasis on the accurate and immediate recording of events.

The primary limitation of our database is that it does not capture all Grade I and II events. However, it accurately captures events requiring a change in the level of care of a patient as well as secondary events that result in permanent end organ disability or death. Additionally, because events in our database are included as they unfold it is possible that events are scored differently in our database than they are scored in retrospective databases, which are scored after the patient has been discharged from the hospital. This is a planned focus of further research, directly comparing our database to the NSQIP database. It is also possible that secondary events that occur after the patient has returned home are not recorded, as a significant number of our patients travel long distances to receive surgical care at our institution. We attempted to address this issue with the telephone survey, but we found that patients could not recall secondary events with the sensitivity necessary to warrant entering events in the database. We are currently investigating other methods to ensure secondary events that occur at other institutions are recorded in our database.

The major difference between the Clavien-Dindo classification and our modification is the criteria for grade 4. In 2004 Dindo³ modified Clavien’s initial 4 grade classification scheme to a 5 grade system, adding “Life-threatening complication requiring ICU management” as grade 4, while our 2001 modification added chronic disability. Because our SSE database tracks deviations from the expected post-operative course we do not capture the normal post-operative recovery, which for some of the procedures performed at our institution can approach 30 days. Though not reflected in the cases randomly selected for our audit, common grade 4 events include anastomotic stenosis following esophageal or gastric resection, diverting ostomy following anastomotic leak, and renal or liver failure. It is possible that we under-estimate the incidence of chronic disability due to our 30-day cutoff for recording SSEs.

This institutional database is complementary to NSQIP and other similar databases. NSQIP’s strength is the risk adjustment it provides while the SSE database’s strength is its comprehensiveness (220 defined events) and its completeness (100% capture); we use both databases in our quality assessment and quality improvement initiatives. We have initiated projects based on NSQIP and then used the SSE database to, in real time, track the impact of our interventions while awaiting final NSQIP results. We were not NSQIP members during 2008, so were unable to compare the SSE database to NSQIP records for selected patients. This is an area of planned study.

The data from this database have provided a clear picture of the secondary events of our specialized patient population, and is routinely reviewed and discussed in our own morbidity and mortality conferences. Due to their comprehensiveness and completeness these data are also accessed and analyzed for nearly every surgical manuscript published by our colleagues. ^15-20 The data allow us to understand outcomes and trends over time and to improve the care at our institution. Several additional efforts to create risk adjusted models to prospectively identify patients likely to experience major post-operative complications post-operatively are currently underway, using the data from our database. The development of such tools will provide invaluable data to help with risk reduction and improve outcomes for our patients.

Table 5.

All Surgical Secondary Events discovered in EMR not reflected in SSE Database.

Grade 1			Grade 2
Name	#		Name	#
Wound Infection		12	Cellulitis		10
Urinary Tract Infection		9	Supraventricular Arrhythmia		6
Cellulitis		8	Urinary Tract Infection		6
Wound Breakdown		8	Wound Infection		5
Pleural Effusion		7	Hypertension		4
Seroma		6	Pleural Effusion		3
Pneumothorax		5	Ventricular Arrhythmia		3
Anemia		4	Allergic Reaction, Anemia, Deep Venous Thrombosis, Myocardial Infarction, Pneumothorax, Pulmonary Embolism, Renal Failure, Urinary Retention		2
Atelectasis		4
Hematoma, Hypertension, Urinary Retention		2
Ascites, Supraventricular Arrhythmia, Wound Infection, Cdiff, Deep Venous Thrombosis, fever, GI Bleed, Hematuria, Neuropathy (Motor), Neuropathy (Sensory), Pneumonitis, Prolonged Air Leak, Renal Failure, Skin Breakdown/Decubitus Ulcer		1	Catheter Related Infection, Dehydration, Empyema, Erectile Dysfunction, Fascial Dehiscence or Evisceration, Flap Failure, Hypocalcemia, Sepsis		1
Grade 3			Grade 4
Name	#		Name	#
Fistula, Pancreatic		1	Facial Nerve Palsy		1
Gastrointestinal Anastomotic Stricture		1
Intra-abdominal Infection or Abscess		1

Cdiff = C. difficile infection

Synopsis.

A single institution, department wide, prospective surgical secondary events database accurately captures and categorizes post-operative surgical secondary events. Such a database can be successfully used for surgical research and quality improvement projects.

Abbreviations

SSEs

Surgical Secondary Events

MSKCC

Memorial Sloan Kettering Cancer Center

EMR

Electronic Medical Record

CI

Confidence Interval

IV

Intra-venous