Impact of Patient Characteristics on Hospital-Level Outcomes Assessment in Congenital Heart Surgery

Abstract

Background

Accurate hospital outcome measures in congenital heart surgery are important to multiple initiatives. While methods have been developed to account for differences in procedural case-mix, characteristics patients bring into the operation that may also vary across hospitals and influence outcome have received less attention. We evaluated the impact of these characteristics in a large cohort.

Methods

Patients undergoing congenital heart surgery at centers participating in The Society of Thoracic Surgeons Congenital Heart Surgery Database (2010 to 2013) with adequate data quality were included. Variation across hospitals in important patient characteristics was examined, and hospital operative mortality rates were compared with and without adjustment for patient characteristics.

Results

Overall, 86 centers (52,224 patients) were included. There was greater than twofold variation across hospitals for nearly all patient characteristics examined. For example, the proportion of a center’s surgical population comprised of neonates ranged from 12.8% to 26.6% across hospitals; the proportion with a non-cardiac anomaly ranged from 0.7% to 5.0%. When hospital mortality rankings were evaluated based on “standard” (adjustment for differences in procedural case-mix alone) versus “full” models (adjustment for both differences in procedural case-mix and patient characteristics), 14.0% changed their ranking for mortality by 20 or greater positions, 34.9% of centers changed which mortality quartile they were classified in, and 14.0% changed their statistical classification (statistically higher, lower, or same-as-expected mortality).

Conclusions

Characteristics of patients undergoing congenital heart surgery vary across centers and impact hospital outcomes assessment. Methods to assess outcomes and relative performance should account for these characteristics.

Accurate measures of hospital performance are important to a variety of stakeholders and ongoing initiatives aiming to assess and improve outcomes across medical and surgical disciplines. These include quality improvement collaboratives focused on identifying and disseminating best practices from top performers, selective contracting of payers with “centers of excellence,” and various wide-scale health care initiatives across the US providing financial incentives to hospitals delivering the highest quality care [1–4]. Measures that accurately reflect hospital performance are critical to the success of all of these initiatives. For example, in order for selective contracting with centers of excellence to actually lead to improved outcomes, the metrics used to designate these centers must accurately reflect those with the best outcomes, providing the highest quality care.

In the field of congenital heart surgery, hospital performance is currently measured and reported as a part of a variety of different initiatives, including collaborative quality improvement activities, designation of congenital heart surgery centers of excellence by various payers, public reporting initiatives recently initiated by The Society of Thoracic Surgeons, and hospital rankings published by US News and World Report [4–6]. It has long been recognized that adjustment for differences in procedural case-mix across hospitals is important to accurate hospital-level outcomes assessment, and this concept is incorporated into most current measures used in the field [7–9]. In addition, it is well known that patient characteristics aside from the type of congenital heart defect and operation performed can also have a significant impact on outcome. For example, several recent studies have demonstrated the impact of factors such as weight at surgery, prematurity, and presence of genetic syndromes and other anomalies on patient outcomes [10–12]. However, the extent to which these factors impact the assessment of hospital-level outcomes has not been investigated to date and remains unclear. In order to influence hospital outcomes assessment these characteristics would not only need to impact outcomes at the patient level, but also vary across hospitals.

The purpose of the present study was to first evaluate variation in patient baseline characteristics across hospitals in children undergoing heart surgery. Second, we assessed the influence of these variables on assessment of hospital level outcomes in a large multicenter cohort.

Material and Methods

Data Source

The Society of Thoracic Surgeons Congenital Heart Surgery Database (STS-CHSD) was used for this study. This database currently represents greater than 90% of all US centers performing congenital heart surgery [13]. Preoperative, operative, and outcomes data are collected on all patients undergoing pediatric and congenital heart surgery at participating centers. Coding is performed by clinicians and trained data managers using the International Pediatric and Congenital Cardiac Code [14]. Data quality and reliability are evaluated through intrinsic verification of data (eg, identification and correction of missing or out of range values and inconsistencies in values across fields), and a formal process of site visits and data audits at approximately 10% of participating institutions annually. The Duke Clinical Research Institute serves as the data warehouse and analysis center for all of the STS National Databases. This study was approved by the Duke University Institutional Review Board and was not considered human subjects research in accordance with the Common Rule (45 CFR 46.102(f)).

Study Population

Patients undergoing cardiac surgery with or without cardiopulmonary bypass at North American centers participating in the STS-CHSD from 2010 to 2013 with data collected using Version 3.0 of the STS Data Collection Form were included (n = 113 centers; 98,885 operations). Data from 27 centers with greater than 10% missing data on key study variables were excluded. From the remaining 86 centers, 11,480 operations occurring after the index (first) cardiovascular operation of each hospitalization were excluded, along with 3,400 operations for patent ductus arteriosus closure in patients weighing 2.5kg or less, 1,302 operations not included in The Society of Thoracic Surgeons-European Association for Cardiothoracic Surgery (STS-CHSD) (STAT) risk adjustment system, 937 operations with missing operative mortality status, and 304 operations with missing data for other key study variables. The final study population included 52,224 operations from 86 centers.

Data Collection and Outcomes

Data collected included patient characteristics and operative data captured in the database using standard definitions [15]. Data regarding patient characteristics included patient age, weight, presence of prematurity (<37 weeks gestation), any non-cardiac congenital anatomic abnormality, chromosomal abnormality, or syndrome, preoperative risk factors as defined in the STS-CHSD, and any prior cardiothoracic surgery [15]. The primary procedure was classified by STAT mortality category [7]. Center characteristics were also collected, including geographic region, and average annual surgical volume of STAT-classified cases during the study period. The primary outcome variable was operative mortality, defined in the STS-CHSD as any mortality occurring inhospital or in any location within 30 days of surgery [16].

Analysis

Patient characteristics were described at the hospital level using standard summary statistics, and variation across hospitals was evaluated. We focused on characteristics known from previous analyses to have a significant impact on patient outcomes [10]. For each hospital, we calculated the percentage rate of each dichotomous variable and the median value of each continuous variable. These hospital-specific percentages and median values were then descriptively summarized by tabulating percentiles (10th, 25th, 50th, 75th, 90th) across hospitals, and magnitude of variation summarized by calculating the ratio of the 90th/10th percentile (these percentiles rather than the minimum and maximum were used to lessen the influence of extreme outliers).

Hospital-level mortality rates and rankings within the cohort were then evaluated and compared using logistic regression models that adjusted for differences across hospitals in procedural case-mix only (“standard” model) versus a “full” model that adjusted for both procedural case-mix and patient characteristics. Adjustment for procedural case-mix was accomplished through adjustment for the primary procedure of the index operation as described previously [17]. Briefly, empirical Bayes methodology was used, which estimates the risk of each primary procedure using actual observed mortality risk, and information borrowed from other procedures in the same STAT category, the latter weighted more heavily in cases of procedures with small sample size [17]. The “full” model also included adjustment for the following patient characteristics: age; any prior cardiovascular operation; any non-cardiac abnormality; any chromosomal abnormality or syndrome; preoperative mechanical circulatory support; shock persisting at time of surgery; preoperative mechanical ventilation, renal dysfunction; neurologic deficit; any other preoperative factor; prematurity (neonates and infants); and weight (neonates and infants). Details of model development and the rationale for inclusion of specific patient characteristics are described elsewhere [17]. Briefly, these characteristics were chosen due to their significant impact on patient outcomes documented in previous studies [10, 17].

Each hospital’s adjusted mortality rate was calculated separately based on the standard and full models by the following formula: adjusted mortality rate = O/E × (overall mortality rate), where “O” denotes the hospital’s observed number of deaths in the study population, “E” denotes the hospital’s expected number of deaths in the study population, according to the model, and “overall mortality rate” represents the aggregate observed mortality rate in the overall cohort, which was equal to 3.70% [18]. For each hospital, a 95% confidence interval (CI) for the adjusted mortality rate was calculated by treating the observed number of deaths as a binomial random variable and treating the expected number of deaths as a constant.

Hospital-level adjusted mortality rates and rankings were then compared based on the data generated from the standard and full models. First we evaluated simple changes in rank position within the cohort for each hospital, depending on whether data from the standard versus full model were used. However, as small changes in rank may not be meaningful from a clinical or policy perspective, additional methods were used to evaluate larger changes [19]. Hospitals were divided into equal sized groups (quartiles for the purposes of this analysis) based on their ranking for mortality, and we evaluated the proportion of hospitals changing mortality quartiles within the cohort when the data from the standard versus full models were used. Finally, we also evaluated changes in hospital’s statistical classification based on the standard versus full models. For this portion of the analysis, each hospital’s point estimate for adjusted mortality and 95% CI were compared with the aggregate mortality rate in the overall cohort. Hospitals were classified as having lower-than-expected mortality if their 95% CI for adjusted mortality fell entirely below the overall aggregate mortality rate, as having higher-than-expected mortality if their 95% CI for adjusted mortality fell entirely above the overall aggregate mortality rate, and as having same-as-expected mortality if their 95% CI for adjusted mortality overlapped the overall aggregate mortality rate. We then evaluated the proportion of hospitals classified in different statistical groups (lower, higher, and same-as-expected mortality) when the standard versus full models were used. All analyses were performed using SAS Version 9.1 (SAS Institute, Inc, Cary, NC) and R Version 3.0.3 software.

Results

Study Population Characteristics

Overall, 86 centers (52,224 patients) were included. The included centers were geographically diverse; 42% South, 26% Midwest, 20% West, and 12% Northeast in the US, and 1 Canadian center. Annual center surgical volume ranged from 24 to 801 operative cases/year (median 139 cases/year).

Hospital Variation in Patient Characteristics

Variation in patient characteristics across hospitals is displayed in Table 1. Even after excluding extreme outliers, there was more than twofold variation between hospitals in the majority of the variables examined. For example, the proportion of a center’s surgical population comprised of neonates varied more than twofold, ranging from 12.8% to 26.6% across hospitals. Weight at surgery was also highly variable, with the proportion of neonates weighing less than 2.5 kg at surgery varying from 5.8% to 20.4% across hospitals. More than twofold variation across hospitals was also seen for several other characteristics, including the proportion of neonates with prematurity, and the proportion of patients who had undergone any previous cardiothoracic operation, had any type of non-cardiac anatomic abnormality, or any of the STS-defined preoperative risk factors.

Table 1.

Variation in Patient Characteristics Across Hospitals

Variable	Hospital Percentiles					Ratio 90th%/10th%
	10th%	25th%	50th%	75th%	90th%
Age at surgery (years)	0.41	0.51	0.63	0.92	1.98	4.8
Neonates	12.8%	17.9%	20.2%	23.8%	26.6%	2.1
Sex, female	42.0%	43.7%	46.2%	48.6%	50.0%	1.2
Weight at surgery (kg)	5.7	6.3	7.0	8.1	11.0	1.9
Neonates with weight <2.5 kg	5.8%	9.8%	12.4%	16.1%	20.4%	3.5
Prematurity in neonates	8.5%	11.7%	14.3%	17.9%	23.5%	2.8
Previous cardiothoracic operation	16.3%	21.1%	26.2%	31.1%	37.6%	2.3
Any non-cardiac anatomic abnormality	0.7%	1.8%	2.7%	3.5%	5.0%	6.9
Any chromosomal abnormality/syndrome	16.4%	19.4%	23.3%	27.0%	31.2%	1.9
STS-defined preoperative factors
Any	12.4%	20.6%	28.7%	36.6%	47.6%	3.8
Mechanical circulatory support	0.0%	0.0%	0.3%	0.6%	1.0%	N/A
Shock at time of surgery	0.0%	0.1%	0.6%	1.3%	2.0%	N/A
Renal failure or dialysis	0.0%	0.4%	0.8%	1.4%	2.7%	N/A
Mechanical ventilation	2.4%	5.5%	7.7%	11.1%	16.4%	7.0
Neurologic deficit	0.0%	0.3%	0.7%	2.0%	3.5%	N/A

The distribution of data across hospitals is displayed, continuous variables displayed as hospital medians. The magnitude of variation across hospitals is described by the ratio of the 90th/10th percentile (excluding extreme outliers).

N/A = cannot be calculated, denominator = zero; STS = The Society of Thoracic Surgeons.

Changes in Hospital Mortality Rankings With and Without Incorporation of Patient Characteristics

Hospital rankings for operative mortality were evaluated based on standard methods which adjusted for differences across hospitals in procedural case-mix alone versus a full model which adjusted for both differences in procedural case-mix and patient characteristics across hospitals. Rank changed by 10 or more positions for 30.2% of hospitals and by 20 or more positions for 14.0% of hospitals (Fig 1). Overall, the median number of rank positions changed was 5 (ranging from no change to a change by 49 rank positions). Hospitals who changed their ranking by 10 or more positions (versus those who did not) had an average annual surgical volume of 144 versus 138 cases/year (p = 0.77), and a surgical population comprised of 27.0% versus 26.6% of patients in STAT categories 4 or 5 (p = 0.43).

Fig 1.

Change in hospital mortality rankings with and without adjustment for patient characteristics. Percent of hospitals changing mortality rankings when standard (adjustment for procedural case-mix only) versus full (adjustment for both procedural case-mix and patient characteristics) models are used.

Changes in Hospital Mortality Categories With and Without Incorporation of Patient Characteristics

Characterization of hospitals in various mortality categories using the 2 different methods was also evaluated. When hospitals were categorized into mortality quartiles within the cohort based on whether the standard versus full model was used, 34.9% of hospitals (30 of 86) changed which mortality quartile they were classified in (Table 2). Eleven hospitals (12.8%) changed from being classified in the highest or lowest quartile to be classified in one of the middle quartiles, and vice versa. These changes are depicted graphically in Figures 2(A) and 2(B).

Table 2.

Change in Hospital Mortality Categories With and Without Adjustment for Patient Characteristics

Type of Change	n = 86 Hospitals
Quartiles
Any change in mortality quartile	30 (34.9%)
Change from highest or lowest quartile to middle quartiles (and vise-versa)	11 (12.8%)
Statistical categories
Any change in statistical classification	12 (14.0%)
Change from statistically higher- or lower- than-expected to same-as-expected	6 (7.0%)
Change from same-as-expected to statistically higher- or lower-than-expected	6 (7.0%)

Number (%) of hospitals changing mortality category classification when the standard (adjustment for procedural case-mix only) versus full (adjustment for both procedural case-mix and patient characteristics) models are used.

See methods section regarding definition of quartiles and statistical categories.

Fig 2.

Hospital mortality rankings and quartiles with and without adjustment for patient characteristics. (A) Displays hospitals ranked in order of increasing mortality based on the standard model, which adjusts for differences in procedural case-mix only. (B) Displays hospitals ranked in order of increasing mortality based on the full model, which adjusts for both differences in procedural case-mix and patient characteristics. Shading of the bars denotes the hospital’s mortality quartile classification based on the standard model, and demonstrates how hospitals change mortality quartiles when the standard versus full model is used.

Changes in statistical classification were also examined (Table 2). Twelve hospitals (14.0%) changed their statistical classification (statistically lower, higher, or same-as-expected mortality) when the standard versus full model was used. Six hospitals (7.0%) changed from being classified as having statistically higher- or lower-than-expected mortality to having same-as-expected mortality, and 6 other hospitals (7.0%) changed from being classified as having same-as-expected mortality to having statistically higher- or lower-than-expected mortality.

Comment

This analysis demonstrates that characteristics of patients undergoing congenital heart surgery vary substantially across hospitals. Methods that focus only on adjustment for differences in procedural case-mix, and do not account for these differences in important patient characteristics that also influence mortality, can lead to wide discrepancies in reported hospital outcomes and incorrect assessments of performance relative to other hospitals.

Case-mix adjustment has been recognized for many years as an important concept in congenital heart surgery outcomes assessment [7–9]. It is necessary due to the wide variety of congenital heart defects, inherent differences in the risk associated with different lesions and their surgical treatment, and known differences across hospitals in the frequency and type of lesions treated. A variety of methods have been developed to account for differences in procedural case-mix, including techniques based on expert opinion (risk adjustment in congenital heart surgery, version 1 [RACHS-1], and Aristotle methodology), and more recent empiric methods (STS-CHSD [STAT] methodology) [7–9]. Most current methods endorsed or utilized by various organizations for assessment and comparison of congenital heart surgery outcomes, including the National Quality Forum, Agency for Healthcare Research and Quality, US News and World Report, various professional societies, and payers, include some type of adjustment or stratification by procedural case-mix [2, 4–6, 20–22].

However, adjustment for baseline patient characteristics and preoperative comorbidities has received less attention. While the RACHS-1 and Aristotle methodologies include the ability to adjust for certain patient characteristics, most metrics currently used in the assessment and ranking of hospital performance in congenital heart surgery do not actually include these types of adjustments and focus on procedural case-mix only [4, 5, 8, 9]. Metrics which do include these variables are often limited to those captured within administrative datasets [20, 21]. More recently, with the development and increasing participation in various clinical registries in the field, more detailed information regarding important patient characteristics has been captured using standard definitions across hospitals [15]. In addition, an increasing number of studies have documented the important impact that these baseline characteristics and comorbidities can have on outcomes at the patient level [10–12]. It has been hypothesized that just as procedural case-mix varies across centers, there is likely variability in these other patient factors as well, but this has not been evaluated to date. Thus the potential influence of patient characteristics on hospital-level outcomes assessment has remained unclear.

The present study documents substantial variation across hospitals in the patient characteristics examined and important differences in assessment and classification of hospital outcomes when these patient characteristics are or are not taken into account. Further, a prior study [17] has demonstrated that statistical models which incorporate patient characteristics versus those which adjust for procedural case-mix alone perform better with a higher C-statistic. Taken together, these results suggest that the variety of current initiatives which rely on accurate assessment and reporting of hospital congenital heart surgery outcomes should utilize metrics that incorporate adjustment for patient characteristics. The importance of accurate outcome measures has been demonstrated in previous studies in other fields where it has been shown that inaccuracies in the designation of high quality or top performing hospitals can result in the failure of policies designed to improve outcomes and quality of care [23]. For example, Dimick and colleagues [23] demonstrated that the restriction of coverage by the Centers for Medicare and Medicaid Services for bariatric surgery to hospitals designated as centers of excellence did not result in improved outcomes, likely because the metrics underlying this designation did not accurately identify hospitals providing the highest quality care.

Limitations

This study included a subset (~70%) of the US centers performing congenital heart surgery [13]. However, the geographic distribution of the included cohort (mirroring that of the overall sample of US hospitals performing congenital heart surgery) and wide range of center volume support the generalizability of our results [13]. In our assessment of variation in patient characteristics across hospitals and impact on outcomes assessment we assumed that differences across hospitals represented “true” variation; however, it is possible that this variation instead represents differences in the completeness or accuracy of coding of variables across hospitals. We attempted to minimize this possibility by excluding hospitals with poor data quality. In addition, the variation seen in our analyses for “concrete” variables such as age and weight at surgery, which are likely less susceptible to miscoding, supports that the variation seen is real. Further, previous studies [24, 25] and database audits support the overall validity of registry data. More detailed audits of the data fields comprising the patient characteristics examined in this study were recently undertaken by the STS-CHSD, and this will provide important information regarding the accuracy of these fields. It is important to note that 24% of hospitals were excluded from the present study due to greater than 10% missing data, and further efforts will be needed to support the capture of complete data regarding patient characteristics, such that accurate outcomes data can be reported for all hospitals. Finally, this analysis focused on general patient characteristics that can have an important impact across different lesions or types of operations. There are other “procedure-specific factors” which may also have an impact on outcome for certain specific operations; for example, the presence of a restrictive atrial septum in patients with hypoplastic left heart syndrome. The recent addition of fields to collect these more detailed procedure-specific data to the STS-CHSD will facilitate analyses to evaluate the impact of these variables on patient and hospital-level outcomes in the future.

Conclusion

This study demonstrates the importance of adjustment for differences in patient characteristics across hospitals in assessment of congenital heart surgery outcomes. Our results suggest that methodology used by federal agencies and other organizations to assess and compare these outcomes should incorporate key patient characteristics in addition to adjustment for procedural case-mix, and support the inclusion of these characteristics into updated STS-CHSD models. Further initiatives are necessary to support and investigate the completeness and accuracy of coding of these important variables across hospitals, as well as to evaluate the impact of more detailed “procedure-specific” factors on hospital-level outcomes assessment.