Initial application in the EACTS and STS Congenital Heart Surgery Databases of an empirically derived methodology of complexity adjustment to evaluate surgical case mix and results

Abstract

OBJECTIVES

Outcomes evaluation is enhanced by assignment of operative procedures to appropriate categories based upon relative average risk. Formal risk modelling is challenging when a large number of operation types exist, including relatively rare procedures. Complexity stratification provides an alternative methodology. We report the initial application in the Congenital Heart Surgery Databases of the Society of Thoracic Surgeons (STS) and the European Association for Cardio-thoracic Surgery (EACTS) of an empirically derived system of complexity adjustment to evaluate surgical case mix and results.

METHODS

Complexity stratification is a method of analysis in which the data are divided into relatively homogeneous groups (called strata). A complexity stratification tool named the STS–EACTS Congenital Heart Surgery Mortality Categories (STAT Mortality Categories) was previously developed based on the analysis of 77 294 operations entered in the Congenital Heart Surgery Databases of EACTS (33 360 operations) and STS (43 934 patients). Procedure-specific mortality rate estimates were calculated using a Bayesian model that adjusted for small denominators. Operations were sorted by increasing risk and grouped into five categories (the STAT Mortality Categories) that were designed to minimize within-category variation and maximize between-category variation. We report here the initial application of this methodology in the EACTS Congenital Heart Surgery Database (47 187 operations performed over 4 years: 2006–09) and the STS Congenital Heart Surgery Database (64 307 operations performed over 4 years: 2006–09).

RESULTS

In the STS Congenital Heart Surgery Database, operations classified as STAT Mortality Categories 1–5 were (1): 17332, (2): 20114, (3): 9494, (4): 14525 and (5): 2842. Discharge mortality was (1): 0.54%, (2): 1.6%, (3): 2.4%, (4): 7.5% and (5): 17.8%. In the EACTS Congenital Heart Surgery Database, operations classified as STAT Mortality Categories 1–5 were (1): 19874, (2): 12196, (3): 5614, (4): 8287 and (5): 1216. Discharge mortality was (1): 0.99%, (2): 2.9%, (3): 5.0%, (4): 10.3% and (5): 25.0%.

CONCLUSIONS

The STAT Mortality Categories facilitate analysis of outcomes across the wide spectrum of distinct congenital heart surgery operations including infrequently performed procedures.

BACKGROUND

In order to perform meaningful multi-institutional outcomes analyses and quality improvement, any database must incorporate the following seven essential elements [1–7]:

1. use of a common language and nomenclature,

2. use of a database with an established uniform core data set for collection of information,

3. incorporation of a mechanism of evaluating case complexity,

4. availability of a mechanism to assure and verify the completeness and accuracy of the data collected,

5. collaboration between medical and surgical subspecialties,

6. standardization of protocols for life-long follow-up and

7. incorporation of strategies for quality assessment and quality improvement.

The analysis of outcomes after surgery requires a reliable method of estimating the risk of adverse events. Formal risk modelling is challenging for rare operations. Complexity stratification provides an alternative methodology that can facilitate the analysis of outcomes of rare operations. Complexity stratification is a method of analysis in which the data are divided into relatively homogeneous groups (called strata). The data are analysed within each stratum.

Complexity stratification tools have proven to be extremely useful in the analysis of outcomes associated with congenital and paediatric cardiac surgery, in part because so many different distinct types of operations are performed [8, 9]. Since 2002, complexity stratification has been used extensively by the following two continental multi-institutional databases:

1. The Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database.

2. The European Association for Cardio-Thoracic Surgery (EACTS) Congenital Heart Surgery Database.

These databases have reported outcomes stratified by complexity using the following methodologies [10]:

1. The Aristotle Basic Complexity Score (ABC Score) and ABC Level (ABC Level) since 2002.

2. The Risk Adjustment for Congenital Heart Surgery-1 methodology (RACHS-1 methodology) since 2006.

The RACHS-1, ABC Score, and ABC Level are based on estimations of risk or complexity derived from processes that started with expert opinion (i.e. subjective probability). Both systems were developed at a time when validated objective clinical data were lacking, and both systems were therefore developed, in a large part, based on expert opinion or subjective probability. With the increasing amount of validated objective clinical data available, in 2009, STS and EACTS published an objective, empirically based index that can be used to identify the statistically estimated risk of in-hospital mortality by procedure and to group procedures into risk categories: The STS–EACTS Congenital Heart Surgery Mortality Score (STAT Mortality Score) and STS–EACTS Congenital Heart Surgery Mortality Categories (STAT Mortality Categories) [11]. We hypothesize that the STAT Mortality Categories will facilitate complexity stratification of operations in the EACTS and STS Congenital Heart Surgery Databases. The purpose of this manuscript is to report the initial application of an empirically derived system of complexity adjustment (STAT Mortality Categories) to evaluate surgical case mix and results.

MATERIAL AND METHODS

STAT Mortality Score and STAT Mortality Categories

The methodology of the development of the STAT Mortality Score and the STAT Mortality Categories was previously published [11]. Briefly, mortality risk was estimated for 148 types of operative procedures using data from 77 294 operations entered into the EACTS Congenital Heart Surgery Database (33 360 operations) and the STS Congenital Heart Surgery Database (43 934 patients) between 2002 and 2007. Procedure-specific mortality rate estimates were calculated using a Bayesian model that adjusted for small denominators. Operations were sorted by increasing risk and grouped into five categories (the STAT Mortality Categories) that were designed to minimize within-category variation and maximize between-category variation.

Databases

The STS Congenital Heart Surgery Database is the largest database in North America dealing with congenital cardiac malformations. It has grown annually since its inception, both in terms of the number of participating centres submitting data and the number of operations analysed. As of 1 November 2011, the STS Congenital Heart Surgery Database currently has 103 participating centres: 100 from the USA and 3 from Canada. The Report of the 2010 STS Congenital Heart Surgery Practice and Manpower Survey, undertaken by the STS Workforce on Congenital Heart Surgery, documented that 125 centres in the USA perform paediatric and congenital heart surgery and 8 centres in Canada perform paediatric and congenital heart surgery [12]. As of July, 2010, the current number of cumulative total operations in the STS Congenital Heart Surgery Database is 164 240. The source of data for this manuscript from the STS Congenital Heart Surgery Database is the Spring 2010 STS Congenital Heart Surgery Database Report [13] that contains data from 85 North American Participants in the STS Congenital Heart Surgery Database, 84 from the USA and 1 from Canada. (An STS Database Participant is either a practice group of cardiothoracic surgeons or an individual cardiothoracic surgeon.)

The EACTS Congenital Heart Surgery Database is the largest database in Europe dealing with congenital cardiac malformations. By July 2010, the EACTS Congenital Heart Surgery Database contained 100 265 operations performed in 84 257 patients, including 20 027 operations in neonates, 32 851 in infants, 40 020 in children and 7367 in adults. As of July, 2010, the EACTS Congenital Heart Surgery Database has 334 centres from 63 countries registered, with 140 active centres from 37 countries submitting data. The source of data for this manuscript from the EACTS Congenital Heart Surgery Database is Bohdan Maruszewski, MD (Chair of the EACTS Congenital Heart Surgery Database) and Zdzislaw Tobota, MD, who point out that ‘The origin of these data is much more disseminated between various economies, cultures, and areas of development, not only in Europe. Actually 20% of data come from Asia, South America and Africa’.

Mortality

All analyses in this manuscript only include index operations (the first cardiac operation of a hospital admission) and exclude operations that have as their primary procedure patent ductus arteriosus (PDA) ligation in patients who weigh ≤2500 g. Mortality is calculated based on the rules published by the STS Congenital Heart Surgery Database Task Force and the Joint EACTS–STS Congenital Database Committee [14, 15].

RESULTS

Table 1 shows data from the STS Congenital Heart Surgery Database for the 4-year time interval of 2006–09, inclusive. Table 2 shows data from the EACTS Congenital Heart Surgery Database for the 4-year time interval of 2006–09, inclusive. Table 3 shows the combined data from data from the STS Congenital Heart Surgery Database and the EACTS Congenital Heart Surgery Database for the 4-year time interval of 2006–09, inclusive.

Table 1:

STS data

STS-EACTS category	Total index operations	Total index operations who died before hospital discharge	Discharge mortality (%)
1	17 332	93	0.54
2	20 114	322	1.6
3	9494	232	2.4
4	14 525	1090	7.5
5	2842	505	17.8
Total cases	64 307	2242	3.5

Table 2:

EACTS data

STS-EACTS category	Total index operations	Total index operations who died before hospital discharge	Discharge mortality (%)
1	19 874	197	0.99
2	12 196	357	2.9
3	5614	282	5.0
4	8287	857	10.3
5	1216	304	25.0
Total cases	47 187	1997	4.2

Table 3:

Combined EACTS data and STS data

STS-EACTS category	Total index operations	Total index operations who died before hospital discharge	Discharge mortality (%)
1	37 206	290	0.78
2	32 310	679	2.1
3	15 108	514	3.4
4	22 812	1947	8.5
5	4058	809	19.9
Total cases	111 494	4239	3.8

DISCUSSION

The outcomes of patients after any form of medical or surgical treatment are influenced by both the nature and severity of illness and the effectiveness of treatment. Any assessment of treatment, therefore, must take into account the characteristics of both the patients and the procedures. These characteristics may be described either as ‘risk factors’ or ‘degrees of complexity’.

In adult cardiac surgery, the analysis of outcomes of hundreds of thousands of patients undergoing coronary artery bypass graft operations and tens of thousands of patients undergoing aortic valve replacement and mitral valve replacement has been used for several purposes [16]:

1. to develop risk-models,

2. to increase the accuracy with which the outcome of a given procedure on a given patient can be predicted and

3. to compare outcomes on non-identical patient groups between centres, surgeons and eras.

This process has involved several components [17]:

1. the refinement of clinical sets of data,

2. establishment of core variables,

3. standardization of definitions,

4. creation of quality controls

5. and the application of rigorous statistical methodology.

Some early efforts relied on Bayesian models, since they are robust with regard to incomplete data, which was an important problem in the early experience of the database. At present, logistic regression models are the principle statistical technique for risk modelling for cardiac surgery for adults with acquired cardiac disease [8, 18]. Initial risk modelling efforts were limited to data concerning patients undergoing isolated coronary artery bypass grafting. At the outset, measures of outcome were confined to mortality prior to discharge from the hospital. Over the past decade, these initial risk models have been recalculated and refined, using data from increasingly large patient populations. Risk models have been developed for isolated cardiac valve replacement procedures, and for combined coronary bypass and cardiac valve replacement procedures. More recently, measures of outcome have been expanded to include aggregate indices of important and durable morbidities, as well as short-term mortality [8].

The challenge of evaluating quality of care in the management of patients with congenital and paediatric cardiac diseases is very different. The version 3.0 minimal data set that are used currently by the EACTS Congenital Heart Surgery Database and the STS Congenital Heart Surgery Database [19] contains 180 individual diagnoses and 257 different procedures. Between 1998 and 2009 inclusive, the EACTS Congenital Heart Surgery Database and the STS Congenital Heart Surgery Database analysed outcomes of 239 197 operations [6]. Many of the individual diagnoses and procedures, while unique and important, are relatively rare. Thus, the application of contemporary conventional risk modelling methodology is possible in only a fraction of the population of interest and provides no objective means of evaluating the performance of centres or surgeons across a diverse and widely variable spectrum of diagnoses and procedures. Recognition of this problem led to the pursuit of new creative approaches to stratification of congenital cardiac patients and procedures with respect to complexity and risk of adverse outcomes [8, 18]. The first step was to develop a nomenclature that facilitated meaningful comparison across institutions [1–6]. The second step was to develop databases that use this standardized nomenclature to track outcomes of operations that treat patients with paediatric and congenital cardiac disease [1–6]. The third step was to develop a means of accounting for differences in ‘case mix’ [2–11, 18]. Nearly simultaneously, two groups undertook the development of tools or systems to address the heterogeneity of groups of patients with congenital cardiac diseases with respect to the challenge and likelihood of successful management, expressed as either the complexity of the operation or the relative risk of mortality. These two methodologies were developed at a time that validated multi-institutional clinical data were lacking and therefore, these two methodologies were based, in a large part, on expert opinion (or subjective probability). One group's efforts led to the development of the RACHS-1 methodology and the other led to the development of the Aristotle Complexity Score. Both of these tools are now utilized by the EACTS Congenital Heart Surgery Database and the STS Congenital Heart Surgery Database. In 2009, when more validated multi-institutional clinical data were available, STS and EACTS published an empirically based tool for analysing mortality associated with paediatric congenital cardiac surgery, based on objective data (STAT Mortality Score and STAT Mortality Categories).

In their initial forms, both the RACHS-1 methodology and Aristotle Complexity Score systems relied heavily upon expert opinion (subjective probability), and to varying degrees on empirical data. Nearly 10 years after the inception of these ideas for stratification of complexity and risk, a considerably larger mass of consistent and quality clinical data exists than was previously available. These data facilitated the creation of the STAT Mortality Score and the STAT Mortality Categories. This manuscript reports the initial application of this methodology in the STS Congenital Heart Surgery Database and the EACTS Congenital Heart Surgery Database.

The purpose of this manuscript was to test the hypothesis that the STAT Mortality Categories will facilitate complexity stratification of operations in the EACTS and STS Congenital Heart Surgery Databases. Combining the data from the two databases allows for a picture of the widespread applicability of this methodology. The purpose of this manuscript was not to compare outcomes between Europe and North America. Because of differences in the distribution of participating institutions in the EACTS Congenital Heart Surgery Database and the STS Congenital Heart Surgery Database, it is not appropriate to use the data in this manuscript to compare results between Europe and North America. The STS Congenital Heart Surgery Database Report used for this manuscript contains aggregate data from two countries: the USA and Canada. The EACTS Congenital Heart Surgery Database has 37 countries submitting data. In comparison with the two countries submitting data to the STS Congenital Heart Surgery Database, the 37 countries submitting data to the EACTS Congenital Heart Surgery Database represent greater economic, cultural, geographic, and developmental variation; in fact, 20% of data in the EACTS Congenital Heart Surgery Database come from Africa, Asia and South America.

One of the goals of this initiative is to increase the global utilization of this methodology. The Japan Congenital Cardiovascular Surgery Database (JCCVSD) has recently been operationalized based on identical nomenclature and database standards as those that are used by EACTS and STS. The JCCVSD began enrolling patients in 2008. By October 2009, 28 centres were participating and over 3000 operations were entered into the JCCVSD in just over one year of data collection. In October 2009, JCCVSD finished their first open recruitment. As a result, a total number of 72 centres have now signed up to participate, representing ∼85% of the professional paediatric cardiac surgical institutes of Japan. The developers of the JCCVSD hope to collaborate with their colleagues across Asia to create an Asian Congenital Heart Surgery Database. JCCVSD will soon begin to report complexity stratified outcomes using the STAT Mortality Categories.

By implementing an objectively derived methodology to stratify operations, the EACTS Congenital Heart Surgery Database and the STS Congenital Heart Surgery Database have increased their ability to report outcomes stratified by operative complexity and potential risk of mortality. Weaknesses of this methodology include the lack of adjustment for patient-specific risk factors, the voluntary nature of the registries, and the potential for inaccurate data in the databases. Both STS and EACTS are working to implement in their Congenital Heart Surgery Databases risk adjustment methodologies that include patient-specific risk factors. Clearly, the results at institutions not participating in these databases might differ from those reported in this manuscript. Both the STS and EACTS Congenital Heart Surgery Database have audit programs in place. Although audit is not performed at all sites, the results of the audits to date have provided reassuring information about the accuracy of the data in these databases [20].

The data reported in this manuscript demonstrate that the STAT Mortality Categories facilitate analysis of outcomes across the wide spectrum of distinct congenital heart surgery operations including infrequently performed procedures. Our group has previously published data documenting that substantial variability in the outcomes of paediatric and congenital cardiac surgery still exists [21]. Even with the use of 5 years of data, because of the relatively small numbers of many operations at most centres, it is not possible to perform statistically meaningful comparisons of mortality between centres after individual operations [21]. Grouping of operations into strata of similar complexity may further facilitate inter-institutional comparisons. These data can aid in quality assessment and quality improvement initiatives. The STAT Mortality Score and STAT Mortality Categories may therefore play an important role in multi-institutional congenital and paediatric cardiac surgical quality assessment and quality improvement initiatives.

CONCLUSIONS

The STAT Mortality Categories facilitate analysis of outcomes across the wide spectrum of distinct congenital heart surgery operations including infrequently performed procedures.

Conflict of interest: none declared.

APPENDIX. CONFERENCE DISCUSSION

Dr Z. Al-Halees (Riyadh, Saudi Arabia): Analysis of outcomes after congenital cardiac surgery is a complex problem and requires a reliable and reproducible process. When the project was started about 10 years ago, it had many loopholes and was subject to many criticisms. However, with continued hard work, international cooperation, and talking a common language, this process seems now more refined and better defined.

Do you think that this is going to be the final product, or should we expect more refinements, and when do you think we will have a final score system that can be used almost universally for assessing the risk in all categories of congenital operations?

Dr Jacobs: I think that the STAT Mortality Score and STAT Mortality Categories, as described today, represent the best methodology current available to stratify the complexity of pediatric and congenital cardiac operations. However, this methodology is certainly not the final product. The next step is going to be to add into this methodology to what we are calling the STAT Morbidity Score and STAT Morbidity Categories. Mortality is only a very small piece of the picture. We need to measure morbidity and include the potential for morbidity in the assessment of operative complexity. Then, we also need to incorporate patient specific variables into this methodology, something that has been done quite well in the Aristotle Comprehensive Complexity Score with the use of procedure-independent factors and procedure-dependent factors. I think these two developments represent the next steps: (1) to incorporate morbidity adjustment and (2) to incorporate patient specific factors including procedure-independent and procedure-dependent variables. We are working on these two advances now. Still, I think that even when we get done with these next two improvements, we will not have the absolute final product, because the science of developing methodology to stratify the complexity of pediatric and congenital cardiac operations is truly an iterative process that hopefully will continue to get better every year.

Dr Al-Halees: The second question, I think you alluded to it in the discussion slides about the centres that do a cluster of high-risk cases and centres that do a cluster of low-risk cases. Do you think with this current system you will be able to reward or make the centres that do the high-risk cluster stand out as being better centres, rather than if one looks at it in a broad way by which they may look not so good?

Dr Jacobs: Zohair, I think that you are describing one of the primary goals of this initiative. I learned this concept from Francois Lacour-Gayet and Martin Elliott when I was training at Great Ormond Street. Francois Lacour-Gayet developed this concept when he started talking about the Aristotle Score. A big part of the rationale for the Aristotle Score, as well as the RACHS-1 methodology and the STAT Score, is to protect or reward centres that are willing to take on the challenge of doing the higher complexity case. These methodologies have multiple advantages, one being that they protect or reward the centres that do the higher complexity cases.

Dr Al-Halees: My last question relates to morbidity. If you want to embark on that project, is it going to be the same process, i.e. is it going to be 10 years down the line, or will it be a shorter process with the experience that was gained from looking at the mortality score?

Dr Jacobs: Under the leadership of my cousin Marshall Jacobs, we are now using data currently in our databases to develop a morbidity classification. Like the STAT Mortality Score and STAT Mortality Categories, this new morbidity classification, the STAT Morbidity Score and STAT Morbidity Categories, will also have a scoring system and 5 categories based on actual objective data. The STAT Morbidity Score and STAT Morbidity Categories will be developed from two components: (1) postoperative length of stay, and (2) the occurrence of any one or more of the following six complications

1. postoperative acute renal failure requiring temporary or permanent dialysis

2. postoperative neurological deficit persisting at discharge,

3. postoperative requirement of a permanent pacemaker

4. postoperative mechanical circulatory support

5. phrenic nerve injury/paralyzed diaphragm,

6. and unplanned reoperation.

Prior to the end of 2011, we anticipate submitting for publication the manuscript that describes the STAT Morbidity Score and STAT Morbidity Categories. Hopefully, we will then operationalize this methodology in our databases in 2012.

Dr Al-Halees: In terms of the Aristotle Comprehensive Score and the data that we are collecting for mortality, it seems to be very comprehensive and very extensive, and if we want to have a universal system that every country can apply, it seems too much. Is there a way that we can reduce the parameters and make it more acceptable universally?

Dr Jacobs: Zohair, I think that is a great question. We are currently working very closely with Francois Lacour Gayet and the developers of Aristotle to create a version of the Aristotle Comprehensive Score that maximizes the retained information but minimizes the data-entry burden, so that it is manageable on a multi-institutional level.

This new version of the Aristotle Comprehensive Score will include an updated Aristotle Basic Score composed of (1) the STAT Mortality Score and STAT Mortality Categories, (2) the STAT Morbidity Score and STAT Morbidity Categories, and (3) an estimation of the technical difficulty of the operation. This updated Aristotle Basic Score will be enhanced with patient specific factors including procedure-independent and procedure-dependent variables. This enhancement will need to be simplified to maximize the retained information and minimize the data-entry burden, perhaps by focusing on common “benchmark” operations.

Within the next one to two years, we are hoping to be able to roll out this new methodology to the STS and the EACTS Congenital Heart Surgery Databases.