Development and Validation of an Agency for Healthcare Research and Quality Indicator for Mortality After Congenital Heart Surgery Harmonized With Risk Adjustment for Congenital Heart Surgery (RACHS‐1) Methodology

Abstract

Background

The National Quality Forum previously approved a quality indicator for mortality after congenital heart surgery developed by the Agency for Healthcare Research and Quality (AHRQ). Several parameters of the validated Risk Adjustment for Congenital Heart Surgery (RACHS‐1) method were included, but others differed. As part of the National Quality Forum endorsement maintenance process, developers were asked to harmonize the 2 methodologies.

Methods and Results

Parameters that were identical between the 2 methods were retained. AHRQ's Healthcare Cost and Utilization Project State Inpatient Databases (SID) 2008 were used to select optimal parameters where differences existed, with a goal to maximize model performance and face validity. Inclusion criteria were not changed and included all discharges for patients <18 years with International Classification of Diseases, Ninth Revision, Clinical Modification procedure codes for congenital heart surgery or nonspecific heart surgery combined with congenital heart disease diagnosis codes. The final model includes procedure risk group, age (0–28 days, 29–90 days, 91–364 days, 1–17 years), low birth weight (500–2499 g), other congenital anomalies (Clinical Classifications Software 217, except for 758.xx), multiple procedures, and transfer‐in status. Among 17 945 eligible cases in the SID 2008, the c statistic for model performance was 0.82. In the SID 2013 validation data set, the c statistic was 0.82. Risk‐adjusted mortality rates by center ranged from 0.9% to 4.1% (5th–95th percentile).

Conclusions

Congenital heart surgery programs can now obtain national benchmarking reports by applying AHRQ Quality Indicator software to hospital administrative data, based on the harmonized RACHS‐1 method, with high discrimination and face validity.

Introduction

Despite recent advances, congenital heart defects remain the most frequent type of structural birth defect, resulting in the highest mortality risk in infancy from birth defects.1 Approximately 25% of those born with congenital heart defects will require cardiac surgery or other immediate intervention to survive.2 While survival after surgical repair of congenital heart defects continues to improve, analyses demonstrate wide variation in outcomes by institution and practitioner.3, 4, 5, 6 Variation among in‐hospital mortality has also been demonstrated across racial/ethnic groups,7, 8, 9, 10 by type of insurance,9, 11 and by institutional volume.6, 12, 13, 14, 15, 16, 17 The Risk Adjustment for Congenital Heart Surgery (RACHS‐1) method for adjusting for baseline differences in patient risk allows meaningful comparisons of in‐hospital mortality among groups of children undergoing surgery for congenital heart disease.18, 19, 20, 21, 22

In 2008, the National Quality Forum (NQF) approved a risk adjustment method developed by the Agency for Healthcare Research and Quality (AHRQ) to facilitate outcome assessment for congenital heart surgery programs. The method uses AHRQ's Healthcare Cost and Utilization Project (HCUP) State Inpatient Databases (SID) to provide a reference population for institution‐level comparisons of in‐hospital mortality among children <18 years of age. While the approved methodology incorporated several features of the RACHS‐1 method, some risk factors included in the AHRQ method differed from those in the validated RACHS‐1 model; these included complex clinical variables that were difficult for clinicians to understand, such as selected major diagnostic classifications and modified diagnosis‐related groups. As part of the NQF endorsement maintenance process, developers of the 2 risk adjustment methods were asked to harmonize the 2 methodologies, to improve face validity and avoid confusion.23 The new harmonized model will replace the risk adjustment for institution‐level outcome assessment for pediatric heart surgery in the AHRQ Quality Indicator Program.

Methods

Risk Adjustment for Congenital Heart Surgery

The RACHS‐1 method was created to adjust for baseline differences in risk when comparing in‐hospital mortality among groups of patients <18 years of age undergoing congenital heart surgery.19, 20 A nationally representative panel of pediatric cardiologists and cardiac surgeons used clinical judgment to place surgical procedures into groups with similar risk for in‐hospital death; the risk categories were then refined empirically using data from the Pediatric Cardiac Care Consortium and 3 statewide hospital discharge databases.20 Rather than predict risk for individual patients, the panel sought to develop a method that would provide accurate comparisons of mortality across groups of patients undergoing congenital heart surgery, using information that did not require excessive primary data collection. The panel considered various coding frameworks, including International Classification of Disease, Ninth Revision, Clinical Modification (ICD‐9‐CM) codes, when developing risk categories, with an intention to place all codes for a specific procedure in the same category, and to eliminate codes that were too nonspecific to assess typical risk for death.

To apply the RACHS‐1 method, cardiac procedures are grouped into 1 of 6 predefined risk categories; category 1 has the lowest risk of in‐hospital mortality and category 6 has the highest (see Data S1). Cases with a combination of cardiac surgical procedures (eg, repair of coarctation of the aorta together with ventricular septal defect closure) are placed in the category of the single highest‐risk procedure. Additional clinical factors included in the risk adjustment model are age at surgery (categorized as ≤30 days, 31 days to <1 year, or 1–17 years), prematurity, the presence of a major noncardiac structural anomaly, and occurrence of combinations of cardiac surgical procedures. The RACHS‐1 risk categories have been validated in a variety of different data sources.18, 21, 22

AHRQ NQF–Approved Method

The Pediatric Quality Indicator (PDI) titled Pediatric Heart Surgery Mortality Rate (PDI 06) was one of a series of quality indicators created by AHRQ to be applied to hospital inpatient data to provide information about the quality of pediatric healthcare (http://www.qualityindicators.ahrq.gov/Modules/pdi_resources.aspx). PDI 06 was originally released in 2002 as an Inpatient Quality Indicator (IQI 10) and included risk adjustment methodology developed by coauthors of this study. Like RACHS‐1, PDI 06 was not intended to predict risk for individual patients. As part of the development of the PDIs module in 2006, PDI 06 was further reviewed and assessed by a panel of 12 pediatric clinicians, who believed that one of the strengths of the indicator was its ability to estimate risks of patients based on their procedure codes by using administrative data. The AHRQ PDIs are publicly distributed, updated, and refined on an annual basis and are used by numerous organizations for a variety of purposes, including comparative reporting, quality improvement and benchmarking, and research.

Unlike the RACHS‐1 methodology, PDI 06 was designed for use with ICD‐9‐CM–coded diagnoses and procedures found in administrative claims and/or billing data. To use PDI 06, congenital heart procedures are grouped into the same 6 risk categories, based on ICD‐9‐CM codes, defined for RACHS‐1. Cases with a combination of cardiac surgical procedures are placed in the category of the single highest‐risk procedure. Additional clinical factors included in the risk adjustment model are age at surgery (categorized as ≤28 days, 29–60 days, 61–90 days, 91 days to <1 year, 1 to <3 years, 3 to <6 years, 6 to <13 years, or 13–17 years), birth weight (<2500 g versus ≥2500 g), the presence of a major noncardiac anomaly (defined by use of the Clinical Classifications Software23 [CCS] 217 [other congenital anomalies] or 221 [respiratory distress syndrome]), selected major diagnostic classifications (5 [diseases and disorders of the circulatory system] and 15 [newborns and other neonates with conditions originating in the perinatal period]) (see Table A.2 in http://www.qualityindicators.ahrq.gov/Downloads/Modules/PDI/V50/Parameter_Estimates_PDI_50_Final.pdf), selected modified diagnosis related groups (503 [cardiac valves and other major cardiothoracic procedures with or without cardiac catheterization] and 508 [major cardiovascular procedures with or without a complication or comorbidity]), and transfer‐in from another hospital (admission type not newborn, and either admission source of “another hospital” or point of origin “transfer from a hospital”).

Harmonization

Developers of the 2 methodologies met to examine differences between the models. The overall goal was not to develop an entirely new model but rather to optimize model performance and face validity using factors already implemented in the existing methods. No changes were made to the 6 surgical risk categories. Clinical factors common to the 2 models were retained. Where differences between the models existed, empirical analyses were used to help the developers determine which variables were most informative. The overall goal of both models—that is, to evaluate mortality differences among groups of patients undergoing congenital heart surgery, adjusting for case complexity differences—was not changed; the models are not intended to predict risk for individual patients.

Data Source: SID

The AHRQ HCUP SID contain information on all inpatient admissions from nearly all community, nonrehabilitation hospitals in participating states, capturing ≈94% of the US population in 2008.24 It incorporates >100 clinical and nonclinical variables for patients of all ages, including demographics, admission and discharge information, expected source of payment, and hospital charges. Up to 30 diagnoses and procedures are coded using the ICD‐9‐CM system. The SID are produced annually and facilitate multistate comparisons and analyses. In 2008, the year used for model harmonization, the combined SID contained ≈5.9 million pediatric discharges from 4083 community, nonrehabilitation hospitals in 42 participating states (www.hcup-us.ahrq.gov/partners.jsp?SID). The SID 2013 was used for model validation; this data set contains discharges from 3897 community, nonrehabilitation hospitals in 42 states plus the District of Columbia.25

Eligibility Criteria

Inclusion criteria were similar for the 2 risk adjustment methods. Identical algorithms were used to identify cases of congenital heart surgery in patients <18 years of age based on ICD‐9‐CM procedure and diagnosis codes. Transcatheter interventions with no cardiopulmonary bypass were excluded, as were heart transplantations, premature infants or neonates ≤30 days of age with patent ductus arteriosus closure as their only cardiac procedure, and cases missing either discharge disposition or age. Heart transplantations were excluded because of substantial variation in risk, and neonatal patent ductus arteriosus closure was excluded because mortality risk is predominantly based on noncardiac conditions. The PDI 06 measure also excluded obstetric admissions (pregnancy, childbirth, purperium), neonates <29 days of age with birth weight <500 g, and discharges resulting in transfer to another hospital. These additional exclusions were incorporated into the harmonized measure. Finally, cases that could not be assigned to a RACHS‐1 risk category were excluded.

Statistical Analysis

Risk factors contained in both the RACHS‐1 and PDI 06 methodologies were retained in the harmonized method. Logistic regression analyses using data from the SID 2008 were used to examine risk factors that differed in the 2 models, either because they were defined differently or because they were included in one model but not the other. Statistical significance was assessed using likelihood ratio tests, comparing models with and without a particular factor. The discrimination of each model for predicting in‐hospital mortality was quantified using the area under the receiver‐operator characteristic curve (c statistic). Results were inspected to select final model components; in general, risk factors were retained to be consistent with other AHRQ PDIs, to increase model discrimination, and to retain clinical face validity.

Model Validation

The final risk adjustment model was applied to the SID 2008 and 2013; the latter database was used only for validation of the model developed with use of 2008 data. Discrimination of the models was assessed using the c statistic and calibrated by using the Hosmer–Lemeshow goodness‐of‐fit test and calibration plots.

Application of the Harmonized Risk Adjustment Method

The risk adjustment model can be used to estimate both risk‐adjusted in‐hospital mortality rates and standardized mortality ratios (SMRs) for individual hospitals relative to the US reference population. For each institution, the SMR is defined as the observed mortality rate divided by the expected mortality rate based on its case mix, as predicted by the risk adjustment model applied to the entire reference population. A generalized estimating equations model was used to account for the correlation among subjects within each hospital. An SMR <1.0 indicates better than expected performance, while an SMR >1.0 indicates poorer than expected performance. The risk‐adjusted in‐hospital mortality rate was computed by using indirect standardization as the institution's SMR multiplied by the mortality rate in the reference population. The SMR has traditionally been used as the institutional summary measure for RACHS‐1, while the risk‐adjusted mortality rate is used in PDI 06; for the harmonized measure, both summary measures are presented for institutions with >3 cases of congenital heart surgery in the SID 2013.

Reliability adjustment was performed on the institution‐specific risk‐adjusted in‐hospital mortality rates.26 To do this, the signal‐to‐noise ratio was defined as the ratio of the variance between all hospitals in the reference population (signal) to the variance within hospitals (noise); the formula is then signal/(signal+noise). To account for the variation in institution‐specific mortality rates as a result of random factors, the hospital‐specific signal‐to‐noise ratio is applied to this rate as an empirical Bayes shrinkage estimator. The resulting “smoothed” rate is the weighted average of the risk‐adjusted rate and the reference population rate.

The AHRQ institutional review board waived the need for informed consent because data were deidentified.

Results

RACHS‐1 risk category was the only factor defined in the same way for both the RACHS‐1 and PDI 06 methodologies; no changes were introduced for the harmonized measure. Presence of a major noncardiac structural anomaly was incorporated by using the PDI 06 definition, after excluding codes for chromosomal abnormalities and respiratory distress syndrome. Empirical analyses were used to define categories for age at surgery, to support the inclusion of birth weight rather than prematurity and to determine cut points. To improve face validity, major diagnostic classifications and modified diagnosis‐related groups were not incorporated in the harmonized measure, while indicators for admissions that were transferred in from another hospital and the occurrence of combinations of congenital heart surgery procedures during the same admission were retained.

The final harmonized risk adjustment model contained the following risk factors: RACHS‐1 risk category (2, 3, 4, 5+6, versus reference category 1; categories 5 and 6 were combined because of the small number of procedures in category 5), age at surgery (≤28 days, 29–90 days, 91–364 days, versus reference category 1–17 years), low birth weight (500–2499 g), presence of a major noncardiac congenital anomaly (CCS 217 excluding 758.xx), multiple congenital heart surgical procedures during the same admission, and admission transfer‐in.

Using the SID 2008, a total of 17 945 cases of congenital heart surgical from 185 hospitals in 38 states (AR, AZ, CA, CO, CT, FL, GA, HI, IA, IL, IN, KS, KY, LA, MA, MD, ME, MI, MN, MO, NC, NE, NJ, NV, NY, OH, OK, OR, PA, SC, SD, TN, TX, UT, VA, WA, WI, WV) met eligibility criteria (Table 1). The final model applied to these cases is shown in Table 2. Two variables were not significant but met harmonization criteria, creating consistency across AHRQ Quality Indicators as well as face validity. The c statistic for model performance was 0.82, indicating excellent discrimination. The Hosmer–Lemeshow goodness‐of‐fit test indicated satisfactory fit of the model to the observed data (P=0.65, Figure 1A).

Table 1.

Description of Patient Cohorts

	Derivation Data Set (HCUP SID 2008)		Validation Data Set (HCUP SID 2013)
	No. of Discharges	Percentage of Discharges	No. of Discharges	Percentage of Discharges
Age at surgery
≤28 d	4355	24.3	3979	23.5
29–90 d	1481	8.3	1175	6.9
91–364 d	4899	27.2	4494	26.5
1–17 y	7219	40.2	7320	43.1
Sex
Male	10 020	55.8	9381	55.3
Female	7925	44.2	7587	44.7
Birth weight 500–2499 g	540	3.0	517	3.1
Prematurity	554	3.1	525	3.1
Major noncardiac structural anomaly	1206	6.7	1317	7.8
RACHS‐1 risk category
1	2194	12.2	2140	12.6
2	6530	36.4	6046	35.6
3	6417	35.8	6371	37.6
4	2054	11.4	1996	11.8
5	32	0.2	18	0.1
6	718	4.0	397	2.3
Multiple (≥2) congenital heart procedures during admission	3668	20.4	3929	23.2
Admission type transfer in	3309	18.4	2678	15.8

RACHS‐1 indicates Risk Adjustment for Congenital Heart Surgery. Source: Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project (HCUP), State Inpatient Databases (SID).24, 25

Table 2.

Harmonized RACHS‐1 Risk Adjustment Model for In‐Hospital Mortality

	Derivation Data Set (HCUP SID 2008)			Validation Data Set (HCUP SID 2013)
	Odds Ratio	95% CI	P Value	Odds Ratio	95% CI	P Value
RACHS‐1 risk category
1	1.00	—	—	1.00	—	—
2	1.09	0.65–1.82	0.75	1.30	0.66–2.55	0.45
3	2.28	1.31–3.94	0.003	2.45	1.27–4.72	0.008
4	2.78	1.57–4.93	<0.001	3.46	1.76–6.80	<0.001
5+6	5.16	2.91–9.14	<0.001	7.10	3.48–14.5	<0.001
Age at surgery
≤28 d	6.28	4.54–8.69	<0.001	6.16	4.35–8.71	<0.001
29–90 d	2.96	2.16–4.07	<0.001	3.32	2.08–5.29	<0.001
91–364 d	1.19	0.89–1.59	0.23	1.80	1.23–2.64	0.002
1–17 y	1.00	—	—	1.00	—	—
Birth weight 500–2499 g	1.96	1.49–2.59	<0.001	2.34	1.74–3.14	<0.001
Major noncardiac structural anomaly	1.27	1.06–1.51	0.008	2.23	1.75–2.84	<0.001
Multiple congenital heart procedures during admission	2.19	1.81–2.66	<0.001	2.15	1.75–2.64	<0.001
Admission type transfer in	0.96	0.76–1.21	0.73	1.10	0.88–1.37	0.04

RACHS‐1 indicates Risk Adjustment for Congenital Heart Surgery.Source: Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project (HCUP), State Inpatient Databases (SID).24, 25

Figure 1.

A, Calibration of harmonized model based on Hosmer–Lemeshow goodness‐of‐fit test (Healthcare Cost and Utilization Project [HCUP] State Inpatient Databases [SID] 2008). B, Calibration of harmonized model based on Hosmer–Lemeshow goodness‐of‐fit test (HCUP SID 2013). Circles represent observed and expected mortality rates within each decile of risk. The solid line represents the linear regression of observed in‐hospital mortality rate vs expected in‐hospital mortality rate. The dashed line represents the situation where observed and expected rates are identical. Source: AHRQ.24, 25

In the SID 2013 validation data set, 16 968 cases of congenital heart surgery from 150 hospitals in 39 states plus the District of Columbia (AR, AZ, CA, CO, CT, DC, FL, GA, HI, IA, IL, IN, KS, KY, LA, MA, MD, MI, MN, MO, NC, ND, NE, NJ, NM, NV, NY, OH, OK, OR, PA, SC, SD, TN, TX, UT, VA, WA, WI, WV) met eligibility criteria (Table 1). RACHS‐1 risk category was the variable with the highest discrimination (c statistic 0.72; odds ratios 2.53 category 2, 6.23 category 3, 15.6 category 4, and 35.3 categories 5+6). The c statistic increased to 0.80 with the addition of age category and to 0.82 for the full model (Table 2). The Hosmer–Lemeshow goodness‐of‐fit test indicated satisfactory fit (P=0.48, Figure 1B). The median risk‐ and reliability‐adjusted in‐hospital mortality rate was 3.0% (5th and 95th percentiles, 0.9% and 4.1%). Median SMR was 0.86 (5th and 95th percentiles, 0.00 and 2.64).

In the SID 2013, discrimination of the harmonized model was equivalent to that for the original AHRQ PDI 06 model (c statistic 0.82) and for the RACHS‐1 model (c statistic 0.82). The P value for the Hosmer–Lemeshow test of calibration was higher for the harmonized model than for either PDI 06 (P=0.09, Figure 2A) or RACHS‐1 (P=0.36, Figure 2B).

Figure 2.

A, Calibration of the Agency for Healthcare Research and Quality (AHRQ) Pediatric Quality Indicator (PDI) 06 model based on Hosmer–Lemeshow goodness‐of‐fit test (Healthcare Cost and Utilization Project [HCUP] State Inpatient Databases [SID] 2013). B, Calibration of Risk Adjustment for Congenital Heart Surgery (RACHS‐1) model based on Hosmer–Lemeshow goodness‐of‐fit test (HCUP SID 2013). Circles represent observed and expected mortality rates within each decile of risk. The solid line represents the linear regression of observed in‐hospital mortality rate vs expected in‐hospital mortality rate. The dashed line represents the situation where observed and expected rates are identical. Source: AHRQ.24, 25

Discussion

Assessment of outcomes for clinical care is a critical component of value‐based purchasing and quality improvement but requires both accurate data from peer organizations and risk adjustment methods to be meaningful. Mandated reporting for hospital payment and other purposes, with regular auditing by fiduciaries and oversight by state health data organizations, have made statewide administrative databases a rich source of information about healthcare delivery. One robust source of administrative data that can be used for health services research is the HCUP project, which is based on the voluntary partnership between AHRQ, state data organizations, and hospital associations (www.hcup-us.ahrq.gov). AHRQ has developed a set of Quality Indicators from the HCUP databases, including the PDIs, and has made tools available to apply the indicators for specific purposes (www.qualityindicators.ahrq.gov).27, 28, 29, 30

The RACHS‐1 method was developed by pediatric cardiologists and congenital heart surgeons as a meaningful summary measure of mortality risk for groups of patients undergoing congenital heart operations, in a manner that would allow institutional comparisons for a significant fraction of cases. RACHS‐1 developers created risk groups to account for the diversity of procedure types and incorporated other clinical risk factors based on empirical evidence from 2 large data sources.20 While RACHS‐1 was created as a general framework for risk, ICD‐9‐CM codes were incorporated from the onset as a part of the original creation and validation, allowing RACHS‐1 to be applied to ICD‐9‐CM–coded data.

NQF uses a rigorous consensus development process to evaluate and endorse standardized healthcare quality measures, incorporating input from diverse healthcare industry stakeholders. This process includes formal calls for quality measures, candidate measure review by standing committees, public and member comment periods, member voting, Consensus Standards Approval Committee endorsement recommendations, ratification of proposed measures by the Board of Directors, and an appeals process. Endorsed measures then undergo annual updating as well as comprehensive maintenance review every 3 years.

In 2008, AHRQ received NQF endorsement for PDI 06, a measure of risk‐adjusted mortality for congenital heart surgery. PDI 06 derived a risk model using the HCUP SID, incorporating clinical risk factors based on ICD‐9‐CM codes. PDI 06 was based on the RACHS‐1 framework but used somewhat different definitions to allow harmonization with other AHRQ measures. As a part of NQF's measure maintenance process, NQF asked AHRQ to harmonize PDI 06 with the original RACHS‐1 methodology, because RACHS‐1 was under consideration by NQF at that time and because the differences between PDI 06 and RACHS‐1 had caused some confusion among stakeholders. The methods and results of this process are described in this report. The resulting methodology was designed specifically to enhance clinical face validity and should be more easily understood by pediatric cardiologists and cardiac surgeons. Using the SID 2013, calibration for the new model was similar to that for both PDI 06 and RACHS‐1 and slightly better. The harmonized RACHS‐1 Pediatric Heart Surgery Mortality Rate measure received full NQF endorsement in 2012.

Because this measure uses clinical information derived from ICD‐9‐CM codes, concerns about the consistency of coding across different institutions have led to criticism of outcome assessment based on ICD‐9‐CM codes, including PDI 06 and RACHS‐1.31, 32, 33, 34, 35, 36 However, critics have frequently failed to recognize that the ICD‐9‐CM and now ICD‐10‐CM code sets are subject to regular augmentation by the Coordination and Maintenance Committee as new procedures are developed and important variations of cardiac anomalies are identified. Critics have also not appreciated that ICD‐9‐CM codes undergo frequent audits related to payment. All AHRQ Quality Indicators are in the process of conversion to ICD‐10‐CM; improvements in coding frameworks should further improve clinical face validity and reduce criticisms based on code limitations. Despite the current limitations, valuable information about variability in outcomes has been derived from ICD‐9‐CM–coded databases.8, 37, 38, 39, 40, 41, 42 Further, use of the HCUP SID for evaluation of outcomes has many strong features. The 2008 and 2013 databases capture ≈95% of the US pediatric population; the databases, therefore, include institutions at which congenital heart surgery is performed, are the largest databases available in the United States, and allow generation of population‐based estimates.

Congenital heart surgery programs can now obtain national benchmarking reports by applying AHRQ Quality Indicators software to their own administrative data, based on the harmonized RACHS‐1 method, with excellent discrimination and high face validity. PDI technical specifications and software are available at: www.qualityindicators.ahrq.gov/Modules/pdi_resources.aspx.

Sources of Funding

This work was funded partially by contract No. HHSA290201200001C between the AHRQ and Battelle Memorial Institute, with a subaward to the University of California Davis School of Medicine.

Disclosures

None.

Supporting information

Data S1. Surgical procedures by Risk Adjustment for Congenital Heart Surgery (RACHS‐1) risk category.

(J Am Heart Assoc. 2016;5:e003028 doi: 10.1161/JAHA.115.003028)