Can Surgeons Adequately Capture Adverse Events Using the Spinal Adverse Events Severity System (SAVES) and OrthoSAVES?

Abstract

Background

Physicians have consistently shown poor adverse-event reporting practices in the literature and yet they have the clinical acumen to properly stratify and appraise these events. The Spine Adverse Events Severity System (SAVES) and Orthopaedic Surgical Adverse Events Severity System (OrthoSAVES) are standardized assessment tools designed to record adverse events in orthopaedic patients. These tools provide a list of prespecified adverse events for users to choose from—an aid that may improve adverse-event reporting by physicians.

Questions/Purposes

The primary objective was to compare surgeons’ adverse-event reporting with reporting by independent clinical reviewers using SAVES Version 2 (SAVES V2) and OrthoSAVES in elective orthopaedic procedures.

Method

This was a 10-week prospective study where SAVES V2 and OrthoSAVES were used by six orthopaedic surgeons and two independent, non-MD clinical reviewers to record adverse events after all elective procedures to the point of patient discharge. Neither surgeons nor reviewers received specific training on adverse-event reporting. Surgeons were aware of the ongoing study, and reported adverse events based on their clinical interactions with the patients. Reviewers recorded adverse events by reviewing clinical notes by surgeons and other healthcare professionals (such as nurses and physiotherapists). Adverse events were graded using the severity-grading system included in SAVES V2 and OrthoSAVES. At discharge, adverse events recorded by surgeons and reviewers were recorded in our database.

Results

Adverse-event data for 164 patients were collected (48 patients who had spine surgery, 51 who had hip surgery, 34 who had knee surgery, and 31 who had shoulder surgery). Overall, 99 adverse events were captured by the reviewers, compared with 14 captured by the surgeons (p < 0.001). Surgeons adequately captured major adverse events, but failed to record minor events that were captured by the reviewers. A total of 93 of 99 (94%) adverse events reported by reviewers required only simple or minor treatment and had no long-term adverse effect. Three patients experienced adverse events that resulted in use of invasive or complex treatment that had a temporary adverse effect on outcome.

Conclusion

Using SAVES V2 and OrthoSAVES, independent reviewers reported more minor adverse events compared with surgeons. The value of third-party reviewers requires further investigation in a detailed cost-benefit analysis.

Level of Evidence

Level II, therapeutic study.

Introduction

Accurate reporting and tracking of adverse events is essential to understanding complications, and subsequently implementing programs to decrease morbidity and mortality. Adverse events can have a substantial economic burden on the healthcare system and a negative effect on overall patient outcomes [21, 29]. Not only is this important regarding the value of care provided to patients but also to overall patient experience and the capacity to improve quality [3, 27]. Although surgeons understand the importance of adverse events, tracking and reporting are weaknesses in the clinical and research environment [2, 25, 43]. A major contributor to these weaknesses lies in the lack of standardized definitions of adverse events [22, 34], with variations of the same adverse events identified across studies [4, 5, 9, 22, 34]. In response to this deficiency, standardized definitions of adverse events have been proposed [16, 17], with the Clavien-Dindo classification being commonly used and reported [7, 10].

Specifically in orthopaedics, Rampersaud et al. developed [32] and validated [33] the Spine Adverse Events Severity System (SAVES), where adverse events are divided in 14 intraoperative and 22 postoperative events, and an “Others” section, with each event graded on a severity scale from 1 to 6. Finally, based on the recorded events, users are prompted to estimate the effect on length of stay. The Orthopaedic Surgical Adverse Events Severity System (OrthoSAVES) is a modification of SAVES that provides events that are applicable to orthopaedics in general as opposed to being spine-specific. The provided list of adverse events in SAVES and OrthoSAVES serves as a prompt for users while also improving convenience of reporting; rather than writing out specific adverse events, users simply check them off on the form. Combined, these features may prove helpful in improving adverse-event reporting by physicians.

The primary objective of this study was to compare the rate of adverse-event reporting by surgeons versus independent clinical reviewers using SAVES and OrthoSAVES in elective orthopaedic procedures.

Methods

This study was a prospective observational analysis of patients undergoing elective orthopaedic spine, hip, knee, and shoulder surgery at a tertiary academic teaching hospital during a 10-week period between June 29, 2015 and September 4, 2015. All patients scheduled for elective surgery, inpatient and outpatient, during this period with the six participating surgeons (GD, PL, PP, EKW, SPK, PEB) were included. Exclusion criteria included patients with trauma or oncologic surgery. This study was approved by the local institutional ethics review board.

During the 10-week study period, 164 patients (48 undergoing spine surgery, 51 having hip surgery, 34 undergoing knee surgery, and 31 having shoulder surgery) underwent elective surgery with participating surgeons. Adverse-event data were reported with SAVES Version 2 (SAVES V2) for patients having spine surgery and OrthoSAVES for patients having hip, knee, or shoulder surgery.

In the hip and knee, arthroplasty accounted for the majority of the procedures. THA (including hip resurfacing) and revisions together accounted for 69% of hip procedures, while knee arthroplasties, including total and unicompartmental, accounted for 88% of knee procedures. In the shoulder, only 10 of 31 procedures (32%) were arthroplasties whereas 19 of 31 (61%) arthroscopic procedures were in the shoulder. This is in contrast to the lower proportion of arthroscopic procedures in the hip (24%) and knee (6%) (Table 1). Among all sites, four revision surgeries were performed: one hip arthroplasty, one hip resurfacing, and two shoulder arthroplasties (Table 2).

Table 1.

Patient demographics and operative details

Variable	Spine (n = 48 patients)	Hip (n = 51 patients)	Knee (n = 34 patients)	Shoulder (n = 31 patients)
Age (years)	56 (15)	49 (17)	59 (14)	54 (13)
Male gender	25 (52%)	29 (57%)	14 (41%)	15 (48%)
BMI, kg/m2	30 (6)	27 (5)	29 (5)	28 (7)
Smokers	10 (21%)	6 (12%)	4 (12%)	10 (32%)
Time of operation (minutes)	211 (128–294)	79 (66–96)	72 (62–77)	78 (62–107)
Outpatient surgery	16 (33%)	13 (26%)	8 (24%)	19 (62%)
Arthroscopic surgery	N/A	12 (24%)	2 (6%)	19 (61%)

Data presented as mean (SD), median (Quartile 1- Quartile 3), or frequency (%); N/A = not applicable

Table 2.

Procedures performed at each site (spine, hip, knee, and shoulder)

Site	Number of procedures
Spine
Discectomy & fusion	22
Microdiscectomy	14
Mixed laminectomy and discectomy & fusion	5
Minimally invasive discectomy & fusion	2
Laminectomy & fusion	2
Discectomy	1
Mixed discectomy and corpectomy & fusion	1
Corpectomy & fusion	1
Hip
THA	22
Hip resurfacing	11
Arthroscopy – chondroosteoplasty head neck junction, acetabular rim trimming	9
Periacetabular osteotomy	4
Arthroscopy – labral chondroosteoplasty, débridement	3
Revision hip arthroplasty and resurfacing	2
Knee
TKA	17
Unicompartmental knee arthroplasty	13
Arthroscopy – ACL reconstruction, medial patellofemoral ligament reconstruction	2
Patellar tendon repair	1
Open débridement and polyethylene liner exchange	1
Shoulder
Arthroscopy – rotator cuff repair, biceps release	16
Total shoulder arthroplasty, including reverse	8
Arthroscopy – labral repair, débridement	3
Open Bankart repair	2
Revision shoulder arthroplasty	2

Adverse-event data were prospectively and independently collected by clinical reviewers (BPC, KG [two first-year medical students]) and surgeons (GD, PL, PP, EKW, SPK, PEB [either attending staff surgeon or orthopaedic surgical resident]). Each team was blinded to the other’s data. SAVES V2 and OrthoSAVES forms were included in the patients’ charts and surgeons recorded adverse events on an ongoing basis based on their clinical interactions and knowledge of their patients. Reviewers thoroughly reviewed the patients’ physical and electronic charts on a daily basis from the time of surgery until discharge to extract adverse-event data. In addition, they had access to all healthcare professionals involved in the patients’ care, except for the surgeons. The reviewers were not involved in the patients’ care and did not interact with the patients. Surgeons and reviewers consulted the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP^®) manual [1] for the definition and criteria for individual adverse events. Adverse events were graded using the six-point classification system included in the SAVES V2 and OrthoSAVES tools [42]. Grade 1 is defined as an event that does not require treatment and has no adverse effect. Grade 2 events require simple or minor treatment, but have no long-term effect on patient outcome. Grades 3 and 4 events require invasive or complex treatment (eg, surgery or monitored bed). Grade 3 events have a temporary (< 6 month) adverse effect while Grade 4 events have a prolonged (> 6 month) adverse effect. Grade 5 events are life or limb threatening or necessitate institutional investigation. Grade 6 is an event resulting in patient death. Minor complications were defined as Grade 1 and Grade 2, while Grade 3 and higher were considered major complications.

Neither reviewers nor surgeons received specific training in using the SAVES V2 and OrthoSAVES tools or adverse-event reporting. If SAVES V2 or OrthoSAVES forms were not completed in full by the surgeons by the time the patient was discharged, the forms were returned to the attending surgeon immediately to be completed. The primary outcome was the number of adverse events recorded by surgeons and reviewers. Secondary outcomes included complication rate and severity of adverse events. Patients were considered to have experienced a complication if at least one adverse event was recorded. Statistical analysis was performed using SAS^® version 9.4 (SAS Institute, Cary, NC, USA). Chi-square and McNemar’s tests were used to compare categorical variables, and the Wilcoxon signed-rank test was used to compare adverse events per patient. Data are presented in median (SD), median (Quartile 1-Quartile 3), or frequency (%) unless otherwise indicated. A two-tailed p value less than 0.05 was considered statistically significant.

Results

Reviewers recorded more total and postoperative adverse events than surgeons (p < 0.001). In total, reviewers recorded 99 adverse events in 57 patients while surgeons recorded 14 adverse events in 12 patients (Table 3). Of the 99 adverse events recorded by reviewers, four were intraoperative and 95 were postoperative. Surgeons recorded two intraoperative and 12 postoperative events. All intraoperative events were in patients having spine procedures and no difference was found between reviewers and surgeons with the sample size available (p = 0.500). At the spine, hip, and knee, reviewers recorded more total adverse events than surgeons (spine, 45 vs 8, p < 0.001; hip, 23 vs 2, p < 0.001; knee, 21 vs 2, p < 0.001), but no difference was found in patients undergoing shoulder procedures with the available sample size (10 vs 2, p = 0.125). Most patients had Grade 1 or 2 adverse events; only three patients had an adverse event more severe than Grade 2. While reviewers recorded more Grades 1 and 2 adverse events than surgeons (Grade 1, 31 vs 2, p < 0.001; Grade 2, 62 vs 8, p < 0.001), there was no difference in capturing Grade 3 adverse events (6 vs 4, p = 0.5). Reviewers and surgeons recorded six and four Grade 3 adverse events, respectively. All four Grade 3 adverse events recorded by the surgeons also were recorded by the reviewers. Reviewers recorded two additional Grade 3 adverse events (pseudoaneurysm and neurologic deterioration) that were not recorded by surgeons. Neurologic deterioration was not recorded by surgeons at all, whereas pseudoaneurysm was recorded, albeit rated as Grade 2 rather than Grade 3 (Table 4).

Table 3.

Number of adverse events per patient

	Frequency		Median (Quartile 1-Quartile 3)		Mean (SD)		p Value
Variable	Reviewer	Surgeon	Reviewer	Surgeon	Reviewer	Surgeon
Total adverse events	99	14	0 (0–1)	0 (0–0)	0.604 (1.08)	0.0854 (0.340)	< 0.0001
Intraoperative	4	2	0 (0–0)	0 (0–0)	0.0244 (0.155)	0.0122 (0.110)	0.500
Postoperative	95	12	0 (0–1)	0 (0–0)	0.579 (1.02)	0.0732 (0.284)	< 0.0001
Comparison by site
Spine	45	8	0 (0–1.5)	0 (0–0)	0.938 (1.39)	0.167 (0.519)	< 0.0001
Hip	23	2	0 (0–1)	0 (0–0)	0.451 (0.673)	0.0390 (0.196)	< 0.0001
Knee	21	2	0 (0–1)	0 (0–0)	0.618 (1.21)	0.0590 (0.239)	0.0005
Shoulder	10	2	0 (0–0)	0 (0–0)	0.323 (0.832)	0.0650 (0.250)	0.125
Severity*
Grade 1	31	2	0 (0–0)	0 (0–0)	0.189 (0.502)	0.0122 (0.110)	< 0.0001
Grade 2	62	8	0 (0–1)	0 (0–0)	0.378 (0.729)	0.0488 (0.216)	< 0.0001
Grade 3	6	4	0 (0–0)	0 (0–0)	0.0366 (0.330)	0.0244 (0.246)	0.500

Reviewer versus surgeon compared with Wilcoxon signed-rank test;* based on severity grading system provided in SAVES and OrthoSAVES.

Table 4.

Summary of reported Grade 3 adverse events

Patient	Procedure	Reviewer-reported	Surgeon-reported
1	Revision decompression (L4-L5) via laminectomy, discectomy, and foraminotomy	Dural tear Cerebrospinal fluid leak/ meningocele Hematoma Neurologic deterioration	Dural tear Cerebrospinal fluid leak/ meningocele Hematoma
2	TKA	Pseudoaneurysm	None
3	TKA	Pulmonary embolism	Pulmonary embolism

Surgeon also recorded pseudoaneurysm, but rated it as Grade 2 rather than Grade 3.

At the spine, hip, and knee, the adverse-event rate was higher when relying on event data from the reviewers compared with from the surgeons (spine, 21/48 [44%] vs 6/48 [13%], p < 0.001; hip, 19/51 [37%] vs 2/51 [4%], p < 0.001; knee, 12/34 [35%] vs 2/34 [6%], p = 0.002), but with the numbers available, we found no differences in patients undergoing shoulder surgery (5/31 [16%] vs 2/31 [7%], p = 0.25) (Fig. 1).

Fig. 1.

The complication rates based on adverse-event data provided by the reviewers and the surgeons are shown. *p < 0.01 compared with surgeons at the corresponding site.

Most commonly reported adverse events differed by type of procedure. Using the reviewers’ data, in the spine, serous drainage and postoperative neuropathic pain accounted for 13% and 11% of all adverse events, respectively. In the hip and knee, the most common adverse event was urinary retention, accounting for 14 of 23 and eight of 21 events, respectively. In the shoulder, airway or ventilation and hypotension each accounted for three of 10 adverse events (Table 5).

Table 5.

Adverse events experienced by patients

Site	Number of events
Spine
Serous drainage	6
Postoperative neuropathic pain	5
Hypertension	4
Emesis	3
Headache	3
Cardiac arrest/failure/arrhythmia	2
Dysphagia	2
Paresthesia	2
Epigastric pain	2
Anemia	2
Anesthesia-related	1
Dural tear	1
Hardware malposition requiring revision	1
Vascular injury	1
Cerebrospinal fluid leak/meningocele	1
Hematoma	1
Neurologic deterioration	1
Spasms	1
Fall	1
Hyponatremia	1
Muscle cramp	1
Urinary retention	1
Hypotension	1
New foot drop	1
Hip
Urinary retention	14
Hypotension	5
Cutaneous injury	2
Airway/ventilation	1
Postoperative neuropathic pain	1
Knee
Urinary retention	8
Hypotension	3
Hypertension	2
Cutaneous injury	2
Decreased level of consciousness	1
Airway/ventilation	1
Pulmonary embolism	1
Serous drainage	1
Anemia	1
Pseudoaneurysm	1
Shoulder
Airway/ventilation	3
Hypotension	3
Fever	2
Anemia	1
Urinary retention	1

On review, 40% of forms were not fully completed at discharge and had to be returned to the attending surgeon. Among the 65 patients for whom forms were incomplete by the time of discharge and thus retrospectively completed by the surgeon, five had adverse events recorded, translating to a complication rate of 7.7%. This is comparable to the adverse-event rate of the other 99 patients for whom surgeons completed the forms prospectively and by the time of discharge. Among these 99 patients, seven (7.1%) had at least one adverse event.

Discussion

Adverse events are becoming increasingly relevant given that the population is aging and postoperative complications increase with age [11, 12, 15, 28, 31, 32]. Although the clinical importance of minor adverse events is controversial [14], the economic burden should be considered. While minor adverse events are less costly individually, they appear with high frequency, thus contributing to more than half of the total costs of adverse events [18, 44]. Furthermore, these events are the most potentially preventable [18], further enhancing the rationale for addressing them. Unfortunately, poor adverse-event reporting by physicians is well-documented in the literature [24, 35, 37]. SAVES and OrthoSAVES may help tackle this issue as the list of adverse events serves as a prompt while also improving convenience of reporting. In the current study, reviewers documented 99 adverse events in patients undergoing elective spine, hip, knee, and shoulder surgery during a 10-week period, of which 93 were Grade 1 or 2 events. In comparison, surgeons reported only 14 adverse events in the same cohort of patients. Another argument for reviewers to prospectively collect adverse event data is the ability to foster timely “point-of-care” interventions. Not only can reviewers serve to improve adverse-event tracking, but they also could inform the clinical team of any patient safety concerns. Whether these benefits are worth the cost of reviewers is unclear and requires further investigation.

Our study had several limitations. First, no formal training was provided to the reviewers or surgeons on the use of SAVES and OrthoSAVES. Thus they learned as the study progressed, which may result in increasing adverse-event capture rates as they became more familiar with the tools. This trend was not observed in our study however. With the first half of the patients, reviewers and surgeons collected 46 and eight adverse events, respectively, while with the second half of the patients, 52 and six adverse events were collected by reviewers and surgeons, respectively. This shows that SAVES and OrthoSAVES are easy-to-use tools that require minimal training. Another limitation is the lack of a gold standard for comparison, which makes it difficult to determine whether surgeons underreported, reviewers overreported, or perhaps a combination of both. However, given that adverse events currently are poorly tracked, it is debatable regarding whether there is a true gold standard and whether such comparisons are useful. Finally, our study was of short duration (10 weeks). A longer duration may lead to better orientation to the tool for surgeons and reviewers and allow for clearer assessment of its long-term viability.

Our finding of poor reporting of adverse events by physicians (in particular, poor reporting of minor adverse events) is congruent with the work of others [24, 35, 37]. In an analysis of 92,547 error and adverse-event reports submitted through an electronic reporting system, only 1.4% of reports were submitted by physicians, compared with 47% submitted by registered nurses [24]. Similarly, in another analysis of 266,244 reports, physicians and nurses submitted 1% and 45% of reports, respectively [35]. Physicians were more likely to report events of higher effect and severity [35, 37], which is also in line with our findings. Schectman and Plews-Ogan [36] surveyed physicians to assess their perceptions of hospital safety and barriers to reporting adverse events. Several important reasons for failure to report these events include being unsure of the reporting mechanism, no actual harm was done to the patient, and the process of reporting being too time consuming. Surveyed physicians believed that allowing electronic reporting of adverse events, clarifying reporting mechanisms, and defining what constitutes an adverse event would likely increase reporting of events [36]. These barriers may help explain the low capture rate of minor adverse events by surgeons in our study. Despite their awareness of the ongoing study, they may not have fully understood what events were reportable; this may be particularly relevant where minor events that cause no harm or adverse effect to the patient are concerned.

In our study, 38% of SAVES V2 and OrthoSAVES forms were not completed by the surgeons by the time of patient discharge. In another study that also relied on surgeons and healthcare staff to collect adverse-event data prospectively using the SAVES V2, the rate of incompletion by the time of discharge was 22% [42]. In both studies, substantial proportions of forms were not completed by the time of patient discharge, requiring the forms to be returned to the attending surgeons to be completed retrospectively. This weakens the quality of adverse-event reporting as the underlying premise is that the data collected prospectively are superior to retrospective data [6, 8, 42]. Our data showed similar complication rates among prospectively and retrospectively collected data. Any interpretation of these values must be done with caution however, as the total number of adverse events recorded by surgeons is small. Furthermore, because attending surgeons usually completed the forms within a few days after discharge, during which time they likely have a strong recollection of the patient’s course in the hospital, it is not expected that adverse-event capture rates should differ importantly.

Among patients having spine surgery, we reported an overall complication rate of 44% using reviewer-collected adverse-event data. This is lower than the complication rate of 87% among patients having spine procedures reported by Street et al. [42]. They also used SAVES V2, but instead of having independent reviewers track adverse events, surgeons, nurses, and other healthcare professionals involved in the patients’ care met on a weekly basis to discuss each patient and record adverse events. Another important difference is that their study included patients treated emergently or for oncologic surgery, whereas we included only patients having elective nononcologic and nontrauma surgery. Emergency cases, as reported by Street et al. [42], were associated with increased intraoperative adverse-event rates and accounted for 95% (19 of 20) of mortalities. Their patients treated with oncologic surgery, electively and emergently, experienced higher rates of postoperative adverse events [42].

In 2004, Dindo et al. [10] developed a classification system for surgical complications, which subsequently has been adapted to several surgical specialties. High interobserver and intraobserver reliabilities have been reported for a version of this classification adapted for orthopaedic surgery [39, 40]. Other instruments, such as the Memorial Sloan Kettering and the Accordion severity grading systems [23, 41], are similar in that severity is graded based on the extent of treatment required and patient outcome. The SAVES and OrthoSAVES tools use a similar severity grading scale, but the key advantage is that specific adverse events common to orthopaedic surgery are listed to prompt users. Perhaps a more modern alternative to these reporting systems relies on automated processes. A study using natural language processing to identify postoperative complications showed higher sensitivity compared with discharge coding with patient safety indicators, albeit with lower specificity [26].

Historically, numerous reports of surgical adverse events were derived retrospectively [13, 19, 20, 30, 38]; our study was prospective. Prior work suggests that retrospective collection of adverse-event data results in underreporting [8, 42]. In a prospective study of adverse events experienced by patients who had undergone emergency spinal surgery for metastatic spine disease, Dea et al. [8] reported an overall complication rate of 76%, higher than previously reported rates of 7% to 39% in retrospective studies [13, 19, 20, 30, 38]. Similarly, Street et al. [42] showed that complication rates, particularly postoperatively, were higher when data were collected prospectively rather than retrospectively. One may argue that since the bulk of adverse-event data reported by the reviewers in our study were extracted from the patients’ charts, there is no difference regarding whether this is done prospectively or retrospectively. However, nurses, physiotherapists, and other healthcare professionals who were knowledgeable about the patients were an important resource for the reviewers that should not be overlooked. This access allowed the reviewers to clarify any issues immediately, which cannot be done if data were collected retrospectively.

We showed that using SAVES V2 and OrthoSAVES in elective orthopaedic surgery, reviewers captured more adverse events compared with surgeons. Although most of the differences were found in reporting of minor complications, these complications can be associated with substantial cumulative costs [18]. However, the cost-benefit of third-party reviewers is unclear. Future studies should aim to provide detailed analysis of the costs of reviewers and the cost savings that they bring by capturing minor complication. Such studies also should evaluate whether identification of these events in real time can assist the clinicians in reducing the frequency of these events or minimizing the costs associated with treating them.