Surgical adverse outcome reporting as part of routine clinical care

Abstract

Background

In The Netherlands, health professionals have created a doctor-driven standardised system to report and analyse adverse outcomes (AO). The aim is to improve healthcare by learning from past experiences. The key elements of this system are (1) an unequivocal definition of an adverse outcome, (2) appropriate contextual information and (3) a three-dimensional hierarchical classification system.

Objectives

First, to assess whether routine doctor-driven AO reporting is feasible. Second, to investigate how doctors can learn from AO reporting and analysis to improve the quality of care.

Methods

Feasibility was assessed by how well doctors reported AO in the surgical department of a Dutch university hospital over a period of 9 years. AO incidence was analysed per patient subgroup and over time, in a time-trend analysis of three equal 3-year periods. AO were analysed case by case and statistically, to learn lessons from past events.

Results

In 19 907 surgical admissions, 9189 AOs were reported: one or more AO in 18.2% of admissions. On average, 55 lessons were learnt each year (in 4.3% of AO). More AO were reported in P3 than P1 (OR 1.39 (1.23–1.57)). Although minor AO increased, fatal AO decreased over time (OR 0.59 (0.45–0.77)).

Conclusions

Doctor-driven AO reporting is shown to be feasible. Lessons can be learnt from case-by-case analyses of individual AO, as well as by statistical analysis of AO groups and subgroups (illustrated by time-trend analysis), thus contributing to the improvement of the quality of care. Moreover, by standardising AO reporting, data can be compared across departments or hospitals, to generate (confidential) mirror information for professionals cooperating in a peer-review setting.

Most studies on medical adverse outcomes (AO) rely on retrieval and reconstruction of events from patients' records by trained reviewers.^1–6 Such approaches have limited effectiveness for quality assessment and improvement. First, retrospective retrieval suffers from the fact that valuable contextual information may be lost when data collection occurs long after the actual medical treatment. Second, older studies often focus on negligence or the type of injury inflicted, and not on learning from past events.^{2 3 5 7}

What is needed, and cannot be delivered by retrospective studies, is a system that encourages healthcare providers to report AOs routinely and then gives immediate and meaningful feedback by which doctors can learn from past events. With these aims in mind, the Dutch Order of Medical Specialists set nationwide standards to create a doctor-driven system of AO reporting that is integrated into routine clinical care. Its key elements are (1) an unambiguous definition of what constitutes an AO, (2) a minimum set of contextual data needed for analysis, and (3) a classification system that organises AOs into clinically and causally relevant groups and subgroups.

The present study elucidates the principles and methods of this system, assesses its advantages and limitations and illustrates its clinical use in the surgical department of a Dutch university hospital.

Patients and methods

A standardised AO reporting system has been in use at the Department of Surgery of the Leiden University Medical Centre since 1997. It is used on all patients who were admitted to, or under the responsibility of, the Department of Surgery, and on patients admitted to non-surgical departments who were operated by a surgeon (or an intern or a resident). Each weekend, a software routine of the Leiden University Medical Centre hospital information system retrieves patients meeting the above surgical criteria who had been discharged in the week that ended 2 weeks before that weekend. The surgical procedures that these patients underwent cover the whole of general surgery, ranging from endocrine, gastrointestinal, oncological, transplant, trauma, to vascular surgery.

Definition of adverse outcome

In The Netherlands, an “adverse outcome” is defined as:

“… an unintended and unwanted event or state occurring during or following medical care, that is so harmful to a patient's health that (adjustment of) treatment is required or that permanent damage results. The adverse outcome may be noted during treatment or in a predefined period after discharge or transfer to another department. The intended results of treatment, the likelihood of the adverse outcome occurring and the presence or absence of a medical error causing it, are irrelevant in identifying an adverse outcome.”

The Dutch definition of an AO is more sensitive than the definitions used in retrospective studies in that those definitions excluded the consequences of the disease itself and/or co-morbidity.^1–3 Using a definition with such exclusion rules in routine reporting would require a doctor to completely understand the cause of an AO and its preventability and/or blame. The Dutch definition obviates such judgement, with the aim of reducing subjectivity and inter-observer variability and thereby facilitating comparisons over time and/or across different settings.

Management of (contextual) information

A special software was designed to support the management of hospital information system and AO data. All the patients identified by the hospital information system search were automatically imported each week into an AO database (Access 2003) with a minimum dataset of contextual information, including name, gender, date of birth, operative procedures performed and the American Society of Anesthesiologists (ASA) classification. Operative procedures were automatically classified with respect to technical complexity, according to the standards of the Association of Surgeons of The Netherlands (on a scale ranging from 1 for simple to 5–7 for complex or difficult surgical procedures).

The occurrence or non-occurrence of all AO was recorded on paper for all patients. Selected AOs were checked and discussed at weekly meetings attended by surgeons, residents and interns and chaired by a staff surgeon. The selection of AOs was based on an algorithm that included patients (1) with three or more AOs, (2) with one or more AOs leading to reoperation, permanent damage and/or death (ie, grade 2 or higher; see below) or (3) with an AO report that seemed likely to contain a lesson for the future. The aim of the above selection was to ensure interesting meetings and thus maximise attendance. Each patient with an AO was presented by the intern/resident who was on the ward when the patient was discharged. The focus of the ensuing discussion was to interpret the underlying causes and thus learn how other such outcomes might be prevented. After the meeting, data forms of all patients, without and with AOs, discussed and non-discussed, are entered into the AO database using the software described above. Patients with one or more AOs are the denominator of the proportion of patients with an AO, the whole patient set being the numerator.

Master classification

AOs were classified according to a special classification system that was designed to match existing (inter)national classification systems as closely as possible. This “master classification” facilitated analysis by allowing AO to be automatically aggregated (by the software) into specific groups and subgroups, The master classification system defines the key characteristics of each AO in terms of the following three dimensions (see Online Appendix A):

1. Type of AO that is its pathology (eg, infection, bleeding)

2. Location, specified by

   • a. region (eg, thorax, abdomen) and/or by

   • b. organ or organ system (eg, lung, liver) and/or by

   • c. tissue

3. Contextual information and potential determinants (eg, medication).

The first dimension—type—is based on the International Classification of Diseases, 10th revision, but with adaptations to accommodate AOs (as opposed to spontaneous disease). The second dimension—location—is based on the Dutch WCC Standard Anatomical Location (http://www.rivm.nl/who-fic/Annuals/Paris26.pdf), developed by WHO. Of the various elements that fall under contextual information dimension, the coding of medication was derived from the Anatomical Therapeutic Chemical classification (http://www.whocc.no/atc_ddd_publications/guidelines/). Each dimension comprises two or more degrees of detail, to allow general and detailed analyses. AOs were graded according to four levels of severity: (1) temporary health disadvantage, recovery without (re)operation; (2) recovery after (re)operation; (3) (probably) permanent damage or function loss; and (4) death.

Analysis

AOs were analysed in two ways: by discussion of individual patients at weekly AO meetings and by regular statistical analysis of patient groups and subgroups.

1. Individual patients were discussed at by means of four structured questions shown below (table 1). The answers given provided (interpretational) information on determinants and preventability. All answers generated since 2003 have been stored in a second database. Lessons learnt can lead to quality improvement on for individual patients and for similar patients in the future.

2. The statistical analysis of the groups and subgroups was performed at regular intervals, at least once a year or whenever deemed necessary—for example, a clinical query or a research question. Logistic regression analysis was used to study the relation between patient mix and operation mix on the one hand, and the incidence of (admissions with) AOs on the other. Patient-mix characteristics were age at admission, gender, and anaesthetic risk profile (high risk being defined as ASA 3 or higher at first operation).^{6 8} The operation mix consisted of various degrees of surgical technical complexity or difficulty.

Table 1.

The four main questions that guide weekly discussions on adverse outcomes

1. What are consequences for patient and care? ▶Severity (four-level scale)—intensity and cost of care

2. What is the complication's main determinant? ▶Targeted disease/co-morbidity/surgery/treatment otherwise

3. In retrospect; could it have been avoided and how? ▶By other treatment choice/execution/pre-care or after-care/… ○If so, then why was this not done? ■Because of pros and cons of alternatives/risk assessment/simply “wrong”

4. Prospectively: what is the lesson learnt? ▶There is none; the complication is “all in the game” ○Then what was the risk?

5. Did the patient know, and how do we know he/she did? ▶Yes, there is a “Lesson”! ○Then what precisely went wrong, and what should be corrected for the future, how, when and by whom?

For the analysis of time trends, the total study period of 9 years was subdivided into three equal periods (P1: 1997–1999, P2: 2000–2002 and P3: 2003–2005), which allowed for a more robust analysis than analysis per year. Logistic regression analysis was carried out for all 18 types of AOs and for each of the four grades of severity. The number of AOs per subset divided by the number of all AOs was taken to be an indicator of a subset's contribution to the overall trend. If the incidence of a specific type of AO changed/increased significantly over time to such an extent that it contributed more than 5% of all AOs, further analysis was performed to identify which subgroups within that category, defined on the second or third dimension, contributed most to the observed time trend.

In all analyses, significance was determined by p values smaller than 0.01, given the large number of patient admissions.

Results

Over the 9-year study period, 19 907 patient admissions met the inclusion criteria for surgical AO reporting. For all these admissions, the absence or occurrence of AOs was documented and verified. Mean patient age at admission was 46.7 (21.9) years, 47.5% of patients were women, 18.5% were ASA 3 or higher and 29.9% underwent “complex” surgery (see table 2). A total of 9189 AOs were reported, with one or more AOs occurring in 18.2% of admissions (table 2).

Table 2.

Data on patient admissions for surgery, operations and adverse outcomes (Leiden University Medical Centre, 1997–2005)

Year	Admissions	Age	Women (%)	ASA at surgery ≥3 (%)	Surgical complexity ≥5 (%)	Admissions with ≥1 adverse outcome
		Mean (SD)				n	%	%
1997	2315	47.3 (20.8)	48.2	14.6	30.3	343	14.7	15.8
1998	2203	46.7 (21.2)	49.2	15.4	26.2	332	15.1
1999	2059	45.9 (21.5)	46.8	15.3	27.3	361	17.5
2000	2008	47.6 (21.4)	46.3	20.7	30.2	368	18.3	17.3
2001	1906	47.1 (21.9)	49.2	21.2	31.0	317	16.6
2002	2196	46.0 (22.7)	46.7	20.7	31.2	373	17.0
2003	2173	45.7 (22.8)	46.3	22.4	29.9	417	19.2	21.3
2004	2467	46.0 (22.2)	46.4	20.8	30.1	517	21.0
2005	2580	47.8 (22.2)	48.4	16.5	32.2	603	23.4
All	19 907	46.7 (21.9)	47.5	18.5	29.9	3631	18.2

Just under half of all AOs (44.5%) were discussed at weekly meetings. Most AOs were deemed to be related to operative treatment (42.5%), followed by non-surgical treatment (29.4%), primary disease (14.5%) and co-morbidity (13.5%). From 2003 to 2005 (P3), 20.9% of all AOs were considered (potentially) avoidable and 4.3% led to specific “lessons learnt”; that is, an average of 55 lessons per year.

Older patients had more AOs, as did high-risk patients and patients undergoing complex surgery (table 3). The overall incidence of reported AOs increased over time (table 3), AO incidence being 39% higher in P3 than in P1, mainly because of an increase in minor AOs (severity category 1, see table 4). The incidence of AOs of severity category 4 (death of the patient) decreased by 40% between 1997 and 2005.

Table 3.

Incidence of registered adverse outcomes in surgical patients over time

Variable	OR (99% CI)*
Year of discharge (reference=1997, with OR 1.00)
1998	1.05 (0.84 to 1.32)
1999	1.29 (1.03 to 1.62)
2000	1.21 (0.97 to 1.52)
2001	1.05 (0.84 to 1.33)
2002	1.13 (0.91 to 1.42)
2003	1.34 (1.08 to 1.67)
2004	1.54 (1.25 to 1.89)
2005	1.71 (1.40 to 2.10)
Age (per life-year)	1.03 (1.03 to 1.03)
Gender (female/male)	0.87 (0.78 to 0.96)
ASA class†
ASA class 3-5	2.42 (2.16 to 2.72)
ASA class missing	1.51 (1.25 to 1.82)
High surgical complexity	1.82 (1.64 to 2.02)

ASA, American Society of Anesthesiologists.

Model fit: Nagelkerke R²=0.171.

^*Significant differences (p<0.01) are indicated in bold.

^†Reference: ASA class 1–2.

Table 4.

Severity of adverse outcome in three periods (1997–1999, 2000–2002 and 2003–2005)

Severity category	2000–2002 vs 1997–1999, OR (99% CI)	2003–2005 vs 1997–1999, OR (99% CI)	Per cent of total number of adverse outcomes
1 Temporary health disadvantage, recovery without (re)operation	0.98 (0.86 to 1.12)	1.40 (1.24 to 1.58)	70.1
2 Recovery after (re)operation	1.13 (0.93 to 1.38)	1.16 (0.96 to 1.41)	21.6
3 (Probably) permanent damage or function loss	1.28 (0.84 to 1.94)	1.36 (0.91 to 2.03)	3.1
4 Death	0.72 (0.55 to 0.94)	0.59 (0.45 to 0.77)	5.1
All adverse outcomes	1.03 (0.90 to 1.17)	1.39 (1.23 to 1.57)	100

In this table, and in tables 5–7, results were adjusted for age, sex, American Society of Anesthesiologists classification and surgical complexity. All ORs are provided with 99% CI (99% CI). Groups of adverse outcomes that changed significantly (p<0.01) over time and contributed more than 5% to the overall incidence are indicated in bold.

Investigation of AO types showed that 7 out of 18 main-type categories occurred more frequently in P2 and/or P3 than in P1. The two categories that occurred most frequently (indicated in bold in table 5) were “inflammation/infection” and “functional disorders” (29% and 23%, respectively; other categories only 7%).

Table 5.

Type of adverse outcome in three periods (1997–1999, 2000–2002 and 2003–2005)^*

Type category	2000–2002 vs 1997–1999, OR (99% CI)	2003–2005 vs 1997–1999, OR (99% CI)	Per cent of total number of adverse outcomes
Rejection/allergy/other immunological reaction	1.68 (0.88 to 3.23)	2.03 (1.09 to 3.76)	1.3
Bleeding/haematoma	0.79 (0.60 to 1.05)	0.90 (0.69 to 1.17)	7.4
Pressure sore	0.88 (0.44 to 1.74)	0.92 (0.48 to 1.77)	1.1
Fistula	2.06 (0.93 to 4.55)	2.41 (1.13 to 5.12)	1.3
Functional disorder	0.97 (0.80 to 1.18)	1.27 (1.06 to 1.52)	22.9
Ischaemia	1.15 (0.61 to 2.16)	1.54 (0.86 to 2.76)	1.3
Symptoms without diagnosis	1.52 (1.03 to 2.25)	2.27 (1.58 to 3.25)	4.4
Injury by mechanical/physical–chemical cause	0.97 (0.71 to 1.34)	1.16 (0.86 to 1.56)	5.0
Necrosis/Infarction	1.63 (1.11 to 2.40)	1.21 (0.82 to 1.79)	3.8
Inflammation/infection	1.05 (0.89 to 1.25)	1.32 (1.13 to 1.54)	28.7
Psychological disturbance	2.03 (1.18 to 3.46)	2.32 (1.39 to 3.89)	2.2
Procedure with specified unintended substandard outcome	0.35 (0.14 to 0.86)	0.48 (0.22 to 1.03)	0.7
Shock	0.94 (0.69 to 1.28)	1.04 (0.77 to 1.40)	5.1
Thrombosis/embolus	1.20 (0.78 to 1.85)	1.30 (0.86 to 1.97)	2.4
Accumulation/leakage of body fluids	1.01 (0.75 to 1.36)	1.26 (0.95 to 1.66)	6.2
Wound healing	1.34 (0.93 to 1.95)	1.38 (0.96 to 1.98)	3.9
Other/non-specified	0.71 (0.47 to 1.05)	0.94 (0.65 to 1.34)	2.2
All adverse outcomes	1.03 (0.90 to 1.17)	1.39 (1.23 to 1.57)	100

^*The main contributing groups of adverse outcomes are analysed further by localisation (tables 6 and 7).

Investigation of AO location showed that AOs of the abdomen, leg (table 6) and circulatory tract (table 7) contributed most to the observed increase in “inflammation/infection”. Investigation of the third dimension—context—revealed that wound infections increased significantly (odds ratio (OR) 1.54 (1.20–1.98) for P3 vs P1) and accounted for 27% of the overall infection increase. Circulatory tract infections, in particular those related with the use of vascular catheters, likewise increased (OR 2.97 (1.51–5.82) in P3 compared with P1) but contributed only 4.4%.

Table 6.

Body regions with infections in three periods (1997–1999, 2000–2002 and 2003–2005)

Region of infection	2000–2002 vs 1997–1999 OR (99% CI)	2003–2005 vs 1997–1999 OR (99% CI)	Per cent of total number of infections
Head	1.26 (0.42 to 3.74)	1.67 (0.61 to 4.56)	1.6
Neck	2.02 (0.56 to 7.36)	3.37 (1.03 to 11.00)	1.5
Back	1.79 (0.19 to 16.79)	1.18 (0.11 to 12.51)	0.3
Thorax	1.02 (0.74 to 1.40)	1.11 (0.82 to 1.50)	18.0
Abdomen	1.19 (0.90 to 1.58)	1.37 (1.06 to 1.79)	30.0
Pelvis	0.91 (0.67 to 1.24)	1.22 (0.92 to 1.60)	20.0
Arm	1.46 (0.55 to 3.87)	1.81 (0.73 to 4.51)	1.9
Leg	1.40 (0.84 to 2.34)	2.33 (1.47 to 3.70)	8.5
Other/non-specified	0.98 (0.70 to 1.38)	1.05 (0.76 to 1.45)	18.2
All infections	1.05 (0.89 to 1.25)	1.32 (1.13 to 1.54)	100

Table 7.

Organ systems with infections in three periods (1997–1999, 2000–2002 and 2003–2005)

Organ system of infection	2000–2002 vs 1997–1999, OR (99% CI)	2003–2005 vs 1997–1999, OR (99% CI)	Per cent of total number of infections
Vestibulocochlear organ	–	0.45 (0.02 to 10.77)	0.1
Circulatory tract	1.24 (0.81 to 1.89)	1.56 (1.05 to 2.31)	11.7
Digestive tract	1.12 (0.70 to 1.81)	0.99 (0.62 to 1.60)	7.0
Endocrine system	–	–	0.1
Genital tract	0.94 (0.26 to 3.43)	1.42 (0.45 to 4.48)	1.0
Skin/subcutis (including breast)	1.72 (0.97 to 3.04)	1.32 (0.73 to 2.36)	5.0
Nervous system	1.14 (0.16 to 8.19)	0.98 (0.14 to 7.08)	0.4
Visual system	1.15 (0.16 to 8.30)	0.49 (0.05 to 5.17)	0.3
Respiratory tract	1.06 (0.75 to 1.50)	1.21 (0.87 to 1.68)	14.9
Urinary tract	0.74 (0.53 to 1.05)	1.08 (0.80 to 1.46)	15.4
Other/non-specified tract	1.23 (0.96 to 1.56)	1.65 (1.32 to 2.06)	44.1
All infections	1.05 (0.89 to 1.25)	1.32 (1.13 to 1.54)	100

The increase in “functional disorders” was mainly explained by a significant increase in functional disorders of the kidney (OR 2.32 (1.52–3.52) for P3 vs P1, and contributing 12.8% of functional disorders), and by medication-related functional disorders (OR 2.32 (1.06–5.06) for P3 vs P1).

Discussion

The present study demonstrates that doctor-driven reporting of AO is feasible alongside routine clinical care over a long period. Even after adjusting for shifts in patient mix and operation mix, it was seen that the number of non-serious AO reported increased rather than decreased, whereas the number of deaths (severity 4 AO) decreased. This suggests that the reporting of AO has become increasingly sensitive over time and that surgical care has become safer with respect to mortality. AO reporting was used to support professionals in assessing and improving their own quality of care in two ways. First, through the weekly AO meetings, surgeons discuss and analyse AOs that their patients suffered case by case. The emphasis lies on ‘What can we learn from this?’ instead of on ‘What have they done wrong?’. The lessons learnt in this way averaged 55 per year in the third period, and applied to many areas ranging from thrombosis prophylaxis in subgroups of patients to technicalities of fracture management at the operating room, and from the appropriate use of antibiotics in trauma patients to timing and preoperative workup of liver transplant patients. Second, at a group level, the aggregation of AOs into master classification groups makes it possible to carry out a statistical analysis of AOs and possible determinants, independent of the weekly case-by-case analyses. As an example, the time-trend analysis presented here demonstrates a significant increase in functional disorders of the kidney, which is most likely related to new immune-modulating drugs used in transplantation and an increase in reported infections. A detailed analysis of the latter has been published in a separate paper.⁹ Infections identified by AO reporting differ from those identified by monitoring of the Centers for Disease Control and Prevention, as AO reporting includes all suspected or proven infections that do not resolve without treatment, that is, that require(d) treatment or have been treated. Apart from the fact that infection monitoring by the Centers for Disease Control and Prevention is not ubiquitous in all Dutch hospitals, it does not routinely provide detailed information on context or potential causes.

Outside the context of the registering department or hospital, doctor-driven AO reporting can also be used for “mutual monitoring”. When professionals from different departments or hospitals cooperate through a (confidential) national AO database, this can provide mirror information to allow doctors to compare their own AO data with anonymous AO data that have been collected by professionals working in similar hospitals or departments. For such a purpose, the system described here has been adopted as the national standard for AO reporting by the Association of Surgeons of The Netherlands. Obviously, additional important issues will need to be addressed, such as adjusting for under-reporting and for more detailed differences in patient and operation mix, to further improve the validity of mirror comparisons.

The Dutch Healthcare Inspectorate analyses the quality of care of all Dutch hospitals annually through national “external” benchmarking of a set of 20 quality performance indicators. One of these indicators is whether medical professionals and hospital departments practice AO reporting. The Inspectorate has, in line with recommendation 6.1 of the Institute of Medicine report “To err is human”ⁱ on peer-review protections, stated that AO data are and will at all times remain the sole property of the medical professionals and will never directly be used for benchmarking. However, if professionals are found to have out-of-range performance indicators, they themselves can use the detailed information provided by the AO reporting system to bring their performance up to standard.

Other research has been published on how to make healthcare safer.^10–12 Chang et al¹³ proposed a standardised terminology and classification for adverse events. The approach presented here differs from Chang et al's in that it is not limited to outcomes that are related to errors in the medical (management) process. By eliminating causality, such as errors, as a criterion for reporting, the “Dutch approach” is more sensitive and has less observer-dependent reporting. In addition, it provides information about all AO and is thus a more relevant source of information because it does not exclude AOs that are not caused by errors. The fact that Dutch AO reporting is integrated into routine clinical practice has advantages over other initiatives that are not rooted in routine patient management, such as the National Confidential Enquiry into Patient Outcome and Death.^{14 15} Not the least of these advantages are the regular plenary discussions among clinicians and the lessons learnt directly each week.

Doctor-driven reporting inevitably has drawbacks as well, the most obvious being underreporting. However, we have shown in an earlier study that, with the overriding focus on learning, underreporting can be kept to a minimum.

The internal monitoring described here, and the mutual monitoring, which is presently part of an initiative in The Netherlands, demonstrate that doctor-driven AO reporting can be a powerful instrument with the potential to prevent AO in patients in the future.