Adjusting for Congenital Heart Surgery Risk Using Administrative data

Abstract

Background:

Congenital heart surgery (CHS) encompasses a heterogenous population of patients and surgeries. Risk standardization models that adjust for patient and procedural characteristics can allow for collective study of these disparate patients and procedures.

Objectives:

We sought to develop a risk adjustment model for CHS using the newly developed Risk Stratification for Congenital Heart Surgery for ICD-10 Administrative Data (RACHS-2) methodology.

Methods:

Within the Kids’ Inpatient Database (KID) 2019, we identified all CHS that could be assigned a RACHS-2 score. Hierarchical logistic regression (clustered on hospital) was used to identify patient and procedural characteristics associated with in-hospital mortality. Model validation was performed using data from 24 State Inpatient Databases (SID) during 2017.

Results:

Of 5,902,538 total weighted hospital discharges in KID 2019, 22,310 pediatric cardiac surgeries were identified and assigned a RACHS-2 score. In-hospital mortality occurred in 543 (2.4%) of cases. Using only RACHS-2, the mortality mode had a C-statistic of 0.81 that improved to 0.83 with the addition of age. A final multivariable model inclusive of RACHS-2, age, payor, and presence of a complex chronic condition outside of congenital heart disease further improved model discrimination to 0.87 (p<0.001). Discrimination in the validation cohort was also very good with C-statistic of 0.83.

Conclusions:

We created and validated a risk adjustment model for CHS that accounts for patient and procedural characteristics associated with in-hospital mortality available in administrative data, including the newly developed RACHS-2. Our risk model will be critical for use in health services research and quality improvement initiatives.

Condensed Abstract:

Risk-adjustment is critical for congenital heart surgery (CHS) where patients and surgeries are heterogenous. The Risk Stratification for Congenital Heart Surgery for ICD-10 Administrative Data (RACHS-2) methodology is a newly developed stratification system for CHS. Our objective was to build a full risk-adjustment model for use in ICD-10 administrative data that incorporates RACHS-2 in addition to other important clinical characteristics. We developed a risk model that adjusts for RACHS-2, age, payor, and presence of a complex chronic condition. Our risk-model has excellent discrimination with C-statistic of 0.87 and 0.83 in the derivation and validation cohorts, respectively.

Introduction

Congenital heart disease (CHD) is the most common type of congenital anomaly, occurring in ~1% of all live births.¹ Despite improvement in survival over time,² CHD remains a significant cause of morbidity, mortality, and expense, with hospital costs estimated at more than $1.4 billion.³ The burden of disease and associated costs highlight the importance of identifying ways to improve the care for this vulnerable population of patients.

Health services and outcomes research and quality improvement initiatives represent important mechanisms to identify deficiencies in care, variability in outcomes across centers, and potential targets for quality improvement for congenital heart surgery (CHS) patients. These types of initiatives are most effective through use of pooled data across centers by way of multicenter clinical or administrative datasets that allow for larger patient samples and more fully represent the spectrum of cardiovascular care. However, accurate analysis and interpretation of data from multicenter registries is contingent upon methods to account for the heterogeneity of patients treated and various CHS procedures across hospitals. Risk adjustment methodology, therefore, has become a critical component of CHS research.

Over the past two decades, various risk adjustment models for CHS have been created both for use in clinical and administrative data. The Risk Adjustment for Congenital Heart Surgery (RACHS-1) methodology created a consensus based tool to classify CHS procedures based upon perceived risk of the surgery and could be used in administrative and clinical data.⁴ While a valuable tool for performing risk stratification, the transition from ICD-9 to ICD-10 in 2015—which included a wholesale redesign of cardiac surgical diagnostic and procedural codes, with limited 1-to-1 correlations— precluded use of RACHS-1 within administrative data after 2015. Most recently, the Risk Adjustment for Congenital Heart Surgery for ICD-10 Administrative data (RACHS-2) stratification methodology was developed and validated for classifying CHS procedures by ICD-10 codes.^5,6 This classification tool was found to predict mortality as well as stratification tools developed for use in clinical registry data.^7,8 However, RACHS-2 incorporated only surgical procedure and age and was not inclusive of other potential patient factors that may predict in-hospital mortality. As was seen in other CHS risk adjustment models, the addition of patient characteristics beyond surgical procedure and age can improve model discrimination.^9,10 To this end, we sought to develop and validate a full CHS risk adjustment model for use in administrative data that incorporates the RACHS-2 stratification system and other patient characteristics associated with in-hospital mortality.

Methods

Study Population

Model development was performed using the 2019 Kids’ Inpatient Database (KID).¹¹ The KID is one of several databases that are part of the Healthcare Cost and Utilization Project (HCUP). The KID was developed through a Federal-State-Industry partnership sponsored by the Agency for Healthcare Research and Quality and is a component of the Healthcare Cost and Utilization Project (HCUP). The KID includes administrative data abstracted by hospitals for billing purposes and is the largest publicly available all-payor pediatric inpatient dataset in the US. The KID is released every 3 years and is a sample of approximately 3 million pediatric discharges from 48 states plus the District of Columbia. The KID includes a sample of pediatric discharges (defined as less than 21 years of age) and samples ≈10% of healthy newborn discharges and ≈80% of other pediatric inpatient discharges. The data set includes a weight variable for each observation so that weighted analysis can produce national estimates of total US discharges for specific diagnoses and procedures. Discharge weights are developed using the American Hospital Association (AHA) reference of community, non-rehabilitation hospitals as the standard. To develop the weights, hospitals are stratified on six characteristics (hospital ownership, bed size, teaching status, rural/urban location, US region, and a stratum for freestanding children’s hospitals). Discharge weights are created within each stratum and reflect the proportion of AHA newborns for newborn discharges and the proportion of (non-newborn) AHA discharges for non-newborn discharges. Data elements available in the KID include patient demographics, primary and secondary diagnoses and procedures, discharge status, length of stay, expected payment source, total hospital charges, comorbidity measures, and hospital characteristics.

Model validation was performed using 2017 data from 24 State Inpatient Databases (SID).¹² The SID is another component of the HCUP set of databases. Data content is similar to variables collected within the KID, but unlike the KID, is not a sample of discharges. HCUP data from each year and data source is subject to standard procedures for ensuring data quality and consistency (https://hcup-us.ahrq.gov/db/quality.pdf).

Study Outcome

The primary outcome was in-hospital mortality defined as any death occurring following a RACHS-2 surgery (see predictor variables below for more details) and occurring prior to hospital discharge. Operative mortality (deaths occurring during hospitalization or within 30 days of the procedure, if discharged prior to 30 days) while often used in CHS, is not available in this dataset, as the data do not follow patients post-discharge. Records were excluded if they were missing data on hospital disposition. These instances were rare comprising only 5 unweighted (8 weighted) encounters.

Patient and Procedural Risk Factors

We considered a number of candidate variables for inclusion in the full risk-adjustment model. These variables were identified among those contained within the HCUP data and those with potential prognostic importance based upon clinical experience or prior literature. RACHS-2 was the primary predictor variable we included in our risk-adjustment model. Details regarding RACHS-2 development and validation have been previously published.^5,6 In brief, RACHS-2 is a recently developed methodology for categorizing CHS within administrative data that uses ICD-10 codes to identify CHS and place them into groups based upon empirically derived risk of mortality (5 groups: category 1=least risk; category 5=most risk). RACHS-2 combined with patient age was found to have discrimination similar to STS and the European Association for Cardio-Thoracic Surgery (EACTS) Congenital Heart Surgery Mortality Categories for risk stratification (STAT Mortality Categories) when the STAT Mortality Categories were applied to linked clinical registry data.

Other predictor variables included patient age (categorized as neonates [<30 days], infants [≥30 days to ≤1 year], toddlers [1–4 years], younger children [5–9 years], older children [10–14 years] and young adults [15–20 years]. Payor was another candidate variable considered for model inclusion (government vs. private insurance vs. other). We also considered patient medical non-cardiac co-morbidities as captured by the pediatric complex conditions classification (CCC) system. Briefly, the CCC classification system was originally developed in 2000 and was created to identify medical conditions that could be “reasonably expected to last at least 12 months (unless death intervenes) and to involve either several different organ systems or 1 organ system severely enough to require specialty pediatric care and probably some period of hospitalization in a tertiary care center.” ^13,14 CCCs include ten broad categories (neuromuscular, cardiovascular, respiratory, renal, gastrointestinal, hematologic or immunologic, metabolic, other congenital or genetic defect, malignancy, and premature and neonatal). CCCs also include a domain indicating that the patient has undergone transplantation and a domain indicating reliance on a medical technology or device.

Statistical Analysis

Baseline characteristics of patients experiencing in-hospital mortality were compared to characteristics of those not experiencing mortality using chi-square tests for categorical variables. Characteristics of the derivation and validation cohorts were also compared using this same methodology to compare baseline similarities.

Hierarchical multivariable logistic regression with backwards elimination was used to identify characteristics predictive of in-hospital mortality while accounting for the clustered nature of the data (patients treated at different hospitals)¹⁵ and consistent with outlined standards for the creation of risk-adjustment models.¹⁶ Hierarchical models allow for assessment of hospital variation in mortality rates after accounting for patient case mix by estimating the log-odds of mortality as a function of demographic and clinical variables (fixed effects) and a random effect for each hospital. For our analysis, we first constructed hierarchical models inclusive of only RACHS-2 score. We then sequentially added clinical variables to assess the incremental change in model discrimination after addition of these variables. A C-statistic was used to assess model discrimination in the derivation and validation cohorts.¹⁷ While changes in C-statistics can be challenging to interpret from a clinical perspective, in general, even small increases in the C-statistic represent significant improvements in models’ ability to predict outcomes, with a C-statistic ≥0.7 and <0.8 indicating acceptable discrimination. A C-statistic ≥0.8 and <0.9 indicating excellent discrimination, and a C-statistic ≥0.9 considered outstanding discrimination.¹⁸ Calibration was assessed in the validation cohort (e.g., agreement between observed outcomes and predictions) and depicted graphically by plotting predicted rates on the x-axis and observed rates on the y-axis. Perfect predictions would be along the 45 degree line. A slop in the calibration plot >1 suggests underestimation in the high risk and overestimation in the low risk. A slope <1 denotes underestimation in the low risk and overestimation in the high risk.¹⁹ The intercept of the calibration plot represents the overall degree of calibration and indicates the extent to which the predictions are systematically too low or too high.²⁰ An intercept of zero suggests good calibration, an intercept >0 denotes an average underestimation, and an intercept <0 denotes an average overestimation.¹⁹ All study analyses were performed with SAS 9.4 (SAS Institute, Cary, NC).

The study was conducted using de-identified administrative data and did not meet criteria for requirement of informed consent. Institutional Review Board approval was not required given that this de-identified dataset did not qualify as human subjects research.

Results

In KID 2019, there are a total of 3,089,283 unweighted and 5,902,538 weighted discharges. Using RACHS-2, we identified 16,630 unweighted encounters for congenital heart surgery from 323 hospitals resulting in a weighted estimate of 22,310 congenital heart surgeries for model development. Of the 323 hospitals, 207 were from hospitals with less than 10 sampled cases. Therefore, 16,242 (97.7%) of the unweighted encounters were from the 116 hospitals with at least 10 sampled cases. We identified 7,467 RACHS-2 congenital heart surgery cases from 99 hospitals within the included SIDs for our validation analysis.

Within the derivation cohort, approximately one-half (51.7%) of the study population was identified as White, 14% were Black, 21.4% Hispanic, 4.1% Asian or Pacific Islander, 0.7% Native American, and 8% of other races. There were slightly more males than females (55.5% vs. 45.5%). Approximately one in five patients were neonates (22.2%) and toddlers (20.2%), one-third were infants, with relatively equal proportion of younger children (9.4%), older children (7.7%), and young adults (8.5%). RACHS-2 categories 1 and 2 were most common, comprising 31.1% and 35.4% of all procedures, respectively. These were performed at 51% and 82% of hospitals, respectively. Only 5% of cases were RACHS-2 category 5, performed at 28% of hospitals. Almost half of patients (46.6%) had a CCC outside of their CHD.

Table 1 provides a comparison of characteristics between the derivation and validation cohorts. While there were some statistically significant differences between baseline characteristics of the derivation and validation cohort, clinically these differences were small. For example, those identifying as Hispanic comprised 21.4% of the derivation cohort (KID) and 15.5% of the validation cohort (SID). Neonates comprised 22.2% of the study population in the KID vs. 25% in the SID.

Table 1.

Comparison of Baseline Characteristics Between the Derivation and Validation Cohort

Variable	SID	KID	p-value
	n=7,467	N=22, 310
Race—no. (%)*
White	3,160 (54.1)	10,551 (51.7)	<0.001
Black	821 (14.1)	2,848 (14)
Hispanic 3	906 (15.5)	4,362 (21.4)
Asian or Pacific Islander	297 (5.1)	842 (4.1)
Native American	71 (1.2)	151 (0.7)
Other	586 (10.0)	1,638 (8.0)
Sex—no. (%)
Male	4,030 (54.0)	12,371 (55.5)	0.026
Female	3,437 (46.0)	9,938 (44.5)
Primary expected payer—no. (%)
Government	3,750 (50.2)	10,728 (48.1)	<0.001
Private	3,407 (45.6)	9,772 (43.8)
Other	310 (4.2)	1,810 (8.1)
Region
Northeast	1,425 (19.1)	3,597 (16.1)	<0.001
Midwest	2,122 (28.4)	5,109 (22.9)
South	2,076 (27.8)	8,652 (38.8)
West	1,844 (24.7)	4,952 (22.2)
Age—no. (%)
Neonates (0–28 days)	1,870 (25.0)	4,943 (22.2)	<0.001
Infants (29–364 days)	2,400 (32.1)	7,158 (32.1)
Toddlers (1–4 years)	1,372 (18.4)	4,500 (20.2)
Younger children (5–9 years)	640 (8.6)	2,103 (9.4)
Older children (10–14 years)	565 (7.6)	1,715 (7.7)
Young adults (15–20 years)	620 (8.3)	1,891 (8.5)
RACHS-2 category —no. (%)
1	2,319 (31.1)	6,938 (31.1)	0.064
2	2,771 (37.1)	7,908 (35.4)
3	1,197 (16)	3,734 (16.7)
4	827 (11.1)	2,605 (11.7)
5	353 (4.7)	1,125 (5.0)
CCC—no. (%)
Neurologic/Neuromuscular
Yes	377 (5.0)	1,191 (5.3)	0.332
Cardiovascular
Yes	6,696 (89.7)	20,438 (91.6)	<0.001
Respiratory
Yes	519 (7.0)	1,814 (8.1)	0.001
Renal and Urologic
Yes	548 (7.3)	1,823 (8.2)	0.022
Gastrointestinal
Yes	986 (13.2)	3,282 (14.7)	0.001
Hematologic/Immunologic
Yes	351 (4.7)	1,071 (4.8)	0.729
Metabolic
Yes	355 (4.8)	1,010 (4.5)	0.418
Other Congenital/Genetic Defect
Yes	1,199 (16.1)	3,812 (17.1)	0.040
Malignancy
Yes	59 (0.8)	190 (0.9)	0.603
Premature/Neonatal
Yes	1,143 (15.3)	2,891 (13.0)	<0.001
Technology Dependence
Yes	1,538 (20.6)	5,179 (23.2)	<0.001
Transplant
Yes	159 (2.1)	647 (2.9)	<0.001
Number of non-cardiovascular CCC
0	4,074 (54.6)	11,906 (53.4)	0.238
1	1,998 (26.8)	6,024 (27.0)
2–3	1,227 (16.4)	3,858 (17.3)
4+	168 (2.2)	522 (2.3)

Abbreviations: RACHS, Risk Adjustment for Congenital Heart Surgery; CCC, complex chronic condition; SID, State Inpatient Database; KID, Kids’ Inpatient Database

^*1,918 KID patients missing data on race; 1,626 SID patients missing data on race

Of all 22,310 RACHS-2 cases in the derivation cohort, 543 (2.4%) resulted in in-hospital death. Table 2 provides a comparison of characteristics for those patients with in-hospital mortality vs. those without. Patients with private insurance were less likely to experience in-hospital mortality compared to those with government or other insurance (1.9% vs. 2.8% vs. 2.9%, p<0.001). Neonates had higher rates of in-hospital mortality than infants, toddlers, younger children, older children, and young adults (6.9% of all neonates, infants 1.4%, toddlers 0.9%, younger children 0.9%, older children 0.9%, young adults 1.4%, p<0.001). As expected, mortality increased with higher RACHS-2 score (category 1 of 0.5%, category 2 of 1.3%, category 3 of 3.1%, category 4 of 5.9%, category 5 of 12.3%, p<0.001). The in-hospital mortality rate incrementally increased with greater number of non-cardiac CCCs. For patients without non-cardiac CCC, mortality was 0.8%, 2.9% for those with one additional CCC, 6.1% for those with 2–3 additional CCC, and 7.5% with four or more CCC.

Table 2.

Baseline patient characteristics of those experiencing in-hospital mortality vs. those who did not

Variable	Total	Survived	Died	p-value
	N=22,310	n=21,766	n=544
Race—no. (%)*
White	10,551 (51.7)	10,329 (51.9)	222 (44.0)	0.001
Black	2,848 (14.0)	2,750 (13.8)	98 (19.4)
Hispanic	4,362 (21.4)	4,246 (21.4)	115 (22.8)
Asian or Pacific Islander	842 (4.1)	824 (4.14)	19 (3.8)
Native American	151 (0.7)	150 (0.8)	1 (0.2)
Other	1,638 (8.0)	1,588 (8.0)	49 (9.7)
Sex—no. (%)
Male	12,371 (55.5)	12,056 (55.3)	315 (57.9)	0.249
Female	9,938 (44.5)	9,709 (44.6)	229 (42.1)
Primary expected payer—no. (%)
Government	10,728 (48.1)	10,424 (47.9)	304 (55.9)	<0.001
Private	9,772 (43.8)	9,585 (44.0)	187 (34.4)
Other	1,810 (8.1)	1,757 (8.1)	52 (9.6)
Region—no. (%)
Northeast	3,597 (16.1)	3,525 (16.2)	72 (13.2)	0.019
Midwest	5,109 (22.9)	4,968 (22.8)	141 (25.9)
South	8,652 (38.8)	8,461 (38.9)	192 (35.3)
West	4,952 (22.2)	4,813 (22.1)	140 (25.7)
Age—no. (%)
Neonates (0–28 days)	4,943 (22.2)	4,602 (21.1)	341 (62.7)	<0.001
Infants (29–364 days)	7,158 (32.1)	7,058 (32.4)	100 (18.4)
Toddlers (1–4 years)	4,500 (20.2)	4,458 (20.5)	42 (7.7)
Younger children (5–9 years)	2,103 (9.4)	2,084 (9.6)	19 (3.5)
Older children (10–14 years)	1,715 (7.7)	1,700 (7.8)	15 (2.8)
Young adults (15–20 years)	1,891 (8.5)	1,864 (8.6)	27 (5.0)
RACHS-2 category —no. (%)
1	6,938 (31.1)	6,906 (31.7)	32 (5.9)	<0.001
2	7,908 (35.4)	7,802 (35.8)	106 (19.5)
3	3,734 (16.7)	3,620 (16.6)	115 (21.1)
4	2,605 (11.7)	2,452 (11.3)	153 (28.1)
5	1,125 (5.0)	987 (4.5)	138 (25.4)
CCC—no. (%)
Neurologic/Neuromuscular
Yes	1,191 (5.3)	1,077 (4.9)	114 (21.0)	<0.001
Cardiovascular
Yes	20,438 (91.6)	19,916 (91.5)	521 (95.8)	<0.001
Respiratory
Yes	1,814 (8.1)	1,692 (7.8)	122 (22.4)	<0.001
Renal and Urologic
Yes	1,823 (8.2)	1,696 (7.8)	126 (23.2)	<0.001
Gastrointestinal
Yes	3,282 (14.7)	3,155 (14.5)	128 (23.5)	<0.001
Hematologic/Immunologic
Yes	1,071 (4.8)	1,009 (4.6)	61 (11.2)	<0.001
Metabolic
Yes	1,010 (4.5)	971 (4.5)	39 (7.2)	0.003
Other congenital/genetic defect
Yes	3,812 (17.1)	3,711 (17.0)	101 (18.6)	0.35
Malignancy
Yes	190 (0.9)	184 (0.8)	7 (1.3)	0.349
Premature/Neonatal
Yes	2,891 (13.0)	2,677 (12.3)	214 (39.3)	<0.001
Technology Dependence
Yes	5,179 (23.2)	4,949 (22.7)	230 (42.3)	<0.001
Transplantation
Yes	647 (2.9)	624 (2.9)	23 (4.2)	0.056
Number of non-cardiovascular CCC—no. (%)
0	11,906 (53.4)	11,813 (54.3)	94 (17.3)	<0.001
1	6,024 (27.0)	5,850 (26.9)	174 (32.0)
2–3	3,858 (17.3)	3,621 (16.6)	237 (43.6)
4+	522 (2.3)	483 (2.2)	39 (7.2)

Abbreviations: RACHS, Risk Adjustment for Congenital Heart Surgery; CCC, complex chronic condition

^*1,918 patients missing data for race (1,879 who died and 40 who did not die)

Three separate multivariable hierarchical logistic regression models were created to evaluate the association between RACHS-2 score and in-hospital mortality accounting for patient and procedural characteristics. A base model adjusting for RACHS-2 category alone yielded a C-statistic of 0.81 [95% confidence interval (CI): 0.78, 0.83) with C-statistic in validation cohort of 0.75(95% CI: 0.71, 0.78). The addition of age to the multivariable model further improved model discrimination, resulting in a model C-statistic of 0.83 (95% CI: 0.81, 0.85) and validation C-statistic of 0.78 (95% CI: 0.75, 0.81). The final multivariable model inclusive of RACHS-2 category, patient age, insurance provider, and complex chronic condition category showed excellent discrimination with a model C-statistic of 0.87 (95% CI: 0.86, 0.89) (Central Illustration), an improvement of 0.04 (95% CI: 0.03, 0.05, p<.001) over the model including RACHS-2 and age. Discrimination in the validation cohort was similar with a C-statistic of 0.83 (95% CI: 0.80, 0.85). The full risk model including model coefficients is included in Supplemental Table 1.

Central Illustration. Predictors of In-Hospital Mortality and Receiver Operating Characteristic Curve for the Final Multivariable Model.

The final multivariable model inclusive of RACHS-2 category, patient age, insurance provider, and complex chronic condition category showed excellent discrimination with a model C-statistic of 0.87 (95% CI: 0.86, 0.89).

A calibration plot in the validation cohort was also constructed. The calibration plot had a slope of 0.98 (standard error [SE] 0.05; p-value [for difference from 1]= 0.68) and an intercept of 0.003 (SE 0.002; p-value [for difference from 0]= 0.28) (Figure 1).

Figure 1. Calibration Plot in the Validation Cohort.

Calibration was assessed in the validation cohort (e.g., agreement between observed outcomes and predictions). The plot had a slope of 0.98 (standard error [SE] 0.05; p-value [for difference from 1]= 0.68) and an intercept of 0.003 (SE 0.002; p-value [for difference from 0]= 0.28).

Discussion

Using two large, administrative datasets, we developed and validated a risk-adjustment model for CHS for use in ICD-10 administrative data that incorporates the newly developed classification system for CHS operations (RACHS-2) in addition to other important patient variables. Our final model, inclusive of demographic, socioeconomic, and clinical variables in addition to RACHS-2 has excellent discrimination and can be used to adjust for patient and procedural characteristics placing patients at higher risk of in-hospital mortality. This tool will be instrumental for CHS research and quality improvement initiatives using ICD-10 administrative data.

Efforts to perform risk adjustment for CHS date back two decades and began with the RACHS-1 methodology and Aristotle Basic Complexity scoring system (ABC).^4,21 Given that case volumes for some CHS are low and given how this could affect empirically derived estimates of risk, these seminal initiatives categorized CHS procedures based upon perceived risk as determined by a committee of national and international experts. For RACHS-1, the group created six risk groups based upon expert opinion, although adjusted groupings when empirically derived risk estimates differed from expert opinion. Empiric data were then used to identify other clinical variables, outside of the specific surgery being performed, that were included in the final multivariable model. In the ABC scoring system, the expert panel divided CHS procedures into four groups based upon expert opinion regarding risk of morbidity, mortality, and technical complexity. These methodologies demonstrated that grouping surgeries based upon risk was a meaningful way to study the collective outcomes of CHS procedures without being limited by the small case numbers for some surgeries. Moreover, these risk-stratification systems demonstrated how, when combined with clinical characteristics, the full risk-adjustment models could be used to evaluate hospital performance²² and to evaluate for other patient characteristics that could be associated with surgical outcomes.^23,24

Following the development of RACHS-1 and ABS, other important CHS risk-adjustment tools were developed. As a combined initiative, the Society of Thoracic Surgeons (STS) and the European Association for Cardiothoracic Surgery (EACTS) engaged in a similar effort aimed at grouping CHS surgeries by risk.^7,8 In contrast to the earlier expert consensus based groupings, the STS-EACTS initiative assessed surgical risk (defined as operative mortality) based largely on empiric data. These ‘STAT’ scores and categories were then used to develop a full risk-adjustment model for the STS-CHS database (STS-CHSD), inclusive of both patient and procedural factors.^9,10 Through this work, the investigators determined that despite the predictive ability of the STAT scores, inclusion of patient-level factors, in addition to procedure-specific factors, improved model predictive ability. The STS-CHSD risk-adjustment model has become an integral part of research and quality improvement efforts using clinical registry data.

The present work represents a critical next step in our efforts to continue with research and quality improvement initiatives that can improve care for CHD patients. Through RACHS-1 and ABC, it became apparent that risk stratification of CHS is possible. The creation of STAT mortality categories established that use of empiric data and a more inclusive array of operations can improve predictive validity when compared to expert consensus. In 2015, the transition from ICD-9 to ICD-10—which included a wholesale redesign of cardiac surgical diagnostic and procedural codes, with limited 1-to-1 correlations—resulted in a need to build upon these efforts and create a risk-stratification system for CHS that could be used for ICD-10 data. RACHS-2 was then empirically created and allowed for risk-stratifying CHS procedures using ICD-10 administrative data including the full complement of surgeries in the STAT Mortality Categories.^5,6 However, recognizing the need to incorporate other important clinical factors, beyond the specific surgery performed, we have created a risk-adjustment model for use in ICD-10 that incorporates RACHS-2 category in addition to other clinical and sociodemographic factors. We believe that our inclusion of both procedural factors (RACHS-category) and other patient characteristics, including medical co-morbidities (presence of a CCC), is of particular importance. Prior work has demonstrated that an increasing number of patients undergoing CHS have medical co-morbidities.²⁵ Creation of a risk-adjustment model that accounts for the increasing complexity of our patient population over time is critically important to ensure that we are accurately characterizing the medical complexity of the patients undergoing CHS. Though we found that surgical procedure, as determined by RACHS-2 category, was the variable most significantly associated with in-hospital mortality, we found that the addition of patient characteristics, including presence of a CCC, significantly improved the ability to predict in-hospital mortality. Our model performance is excellent, with C-statistics of 0.87 and 0.83 in the derivation and validation cohorts, respectively, a predictive ability similar to the predictive ability of the STS-CHSD clinical risk models^9,10.

The field of CHD encompasses a broad spectrum of disease ranging from relatively simple defects and surgeries of low operative risk, to complex CHD requiring repair or palliation that introduces heterogeneity and high risk. Given this heterogeneity, evaluating differences in outcomes between centers can be challenging. The implementation of various risk-adjustment strategies has facilitated these comparisons and allowed the CHD community to identify variations in outcomes between surgeons, centers^26–28 and work towards improving care when variability is found. At present a risk-adjustment model for use in clinical registry data is readily available^9,10 but a full risk-adjustment model for use in ICD-10 administrative data is not. While CHD research using clinical registry data provides critical insights into the care and outcomes of these patients, administrative data provides information that is directly relevant to policy initiatives, is more nationally comprehensive than clinical data and thus critical to the understanding of population-based health. While clinical registries like the STS-CHSD encompass data from many centers nationally performing CHS, participation is voluntary and not all centers performing CHS contribute data. Low volume centers are particularly lacking. In fact, prior studies using administrative data have identified over 150 centers performing CHS in contrast to the ~115 centers contributing data to the STS and in contrast to the 323 hospitals that were identified in our study.^26,29 Moreover, clinical registry data may not be as readily accessible as administrative data, owing to the high costs and time associated with clinical registry multisite data access. For these reasons, our work represents an important contribution to the CHS community, particularly for health services researchers and hospital administration using administrative data sources.

Limitations

Our study has limitations. First, given the lack of direct patient identifiers, we were unable to link this de-identified administrative data to clinical data as was done in prior work.⁵ However, given that RACHS-2 has already been validated and case capture between administrative and clinical registry data already compared as part of prior work⁵, we have no reason to be believe that case capture for the current study would be insufficient. Second, as was noted as a limitation in the initial work detailing development of RACHS-2, not all ICD-10 codes map with precision and RACHS-2 does not always differentiate subtypes of specific operations (Ross vs. Ross-Konno; Fontan vs. Fontan revision, etc.). While this limitation should be noted by investigators using RACHS-2, the intent of RACHS-2 was to capture CHS procedures using ICD-10 administrative data and stratify them by predicted risk of mortality. The excellent discrimination of RACHS-2 suggests that the aggregation of procedural subtypes did not negatively impact the model’s predictive ability. Third, our study accounts for clinical characteristics measurable within administrative data and lumped into large groupings. This does not account for clinically meaningful but difficult to measure differences (such as valve morphology), or even differences between genetic syndromes. Since it is known that some centers make decisions to accept or not accept patients based on these unmeasured confounders, our model should not be used to rank centers. Lastly, our study accounts for some social determinants of health including payor. There is growing evidence that patient outcomes also differ by the neighborhoods in which patients live in ways not sufficiently captured by hospital.³⁰ Further studies are needed to explore the impacts of neighborhood on outcomes.

Conclusions

Using a large multicenter database, we created a risk-adjustment model that can be used for ICD-10 administrative data and one that incorporates empirically derived estimates of risk. Our model has excellent predictive ability that is comparable to risk models using clinical registry data. Our work can be used to study the population-based outcomes of patients with a spectrum of CHD and undergoing all types of CHS operations. By adjusting for patient clinical factors and comorbidities, social determinants of health, and risk of the specific CHS being performed, our model can be applied to study outcomes for these patients and identify targets for quality or process improvement.

Core Clinical Competencies.

Competency in Systems-Based Practice:

This validated risk-adjustment model accounts for characteristics including patient age, co-morbid conditions, socioeconomic status, and type of congenital heart surgery being performed. Applying this risk model to administrative data allows for equitable between hospital comparisons for those engaged in health services research and quality improvement initiatives.

Translational Outlook 1:

Future studies applying this risk model are needed to understand factors contributing to differences in outcomes across centers and how best to improve care across all centers.

Abbreviations:

CHD

congenital heart disease

CHS

congenital heart surgery

ICD

International Classification of Diseases

RACHS

Risk Adjustment for Congenital Heart Surgery

RACHS-2

Risk Adjustment for Congenital Heart Surgery for ICD-10

KID

Kids’ Inpatient Database

SID

State Inpatient Database

AHA

American Hospital Association

HCUP

Healthcare Cost and Utilization Project

STAT Mortality Categories

Society of Thoracic Surgeons and the European Association for Cardio-Thoracic Surgery Congenital Heart Surgery Mortality Categories

CCC

complex chronic condition

CI

confidence interval

SE

standard error

ABC

Aristotle Basic Complexity

STS-CHSD

Society of Thoracic Surgeons Congenital Heart Surgery Database