Validation of a Simple Score to Determine Risk of Hospital Mortality after the Norwood Procedure

Abstract

Objective

The ability to quantify patient-specific hospital mortality risk prior to the Norwood procedure remains elusive. This study aimed to develop an accurate and clinically feasible score to assess the risk of hospital mortality in neonates undergoing the Norwood procedure.

Methods

All patients (n=549) in the publically available Pediatric Heart Network Single Ventricle Reconstruction trial database were included in the analysis. Patients were randomly divided into a derivation (75%) and validation (25%) cohort. Pre-operative factors found to be associated with mortality upon univariable analysis (p<0.2) were included in the logistic regression model. The score was derived by including variables independently associated with mortality (p<0.05).

Results

A 20-point score using 6 variables (birthweight, clinical syndrome or abnormal karyotype, surgeon Norwood volume/year, anatomic subtype, ascending aorta size, and obstructed pulmonary venous return) was developed using relative magnitudes of the covariates’ odds ratio. The score was then tested in the validation cohort. In weighted regression analysis, model predicted risk of mortality correlated closely with actual rates of mortality in the derivation (R²=0.87, p<0.01) and validation cohorts (R²=0.82, p<0.01). Patients were classified as low (score 0–5), medium (6–10), or high risk (>10). Mortality differed significantly between risk groups in both the derivation (6% vs. 22% vs. 77%, p<0.01) and validation (4% vs. 30% vs. 53%, p<0.01) cohorts.

Conclusion

This mortality score is accurate in determining risk of hospital mortality in neonates undergoing planned Norwood operations. The score has the potential to be used in clinical practice to aid in risk assessment prior to surgery.

Introduction

Pre-operative risk factors for hospital mortality after the Norwood procedure for patients with hypoplastic left heart syndrome (HLHS) have been investigated by many groups using data from both single and multi-center investigations. Multiple risk factors have been identified, including low birth weight, additional congenital abnormalities, and presence of a syndromic or genetic abnormality.^1–16 Center surgical volume has also been shown to increase the risk of mortality after the Norwood procedure.^{2, 17, 18} Despite these investigations, the ability to quantify patient-specific mortality risk prior to the Norwood procedure remains elusive.

The NIH/NHLBI Pediatric Heart Network Single Ventricle Reconstruction (SVR) trial prospectively collected preoperative, operative, and postoperative data in the largest cohort of newborn infants with HLHS and other single right ventricle anomalies to date. The publically available SVR trial dataset provides a unique opportunity to quantify the associations of pre-operative patient-related risk factors to hospital mortality after the Norwood procedure. The objective of this study was to design and validate a clinically derived Norwood mortality risk score that could be used to assess pre-operative risk of hospital mortality in neonates with HLHS undergoing the Norwood procedure.

Methods

A retrospective analysis was performed using data obtained from the NIH/NHLBI Pediatric Heart Network Single Ventricle Reconstruction trial publically available dataset. Details of the SVR trial design have been previously published.^{19, 20} In brief, patients were eligible if they had a diagnosis of HLHS or other single right ventricle anomaly and a planned Norwood procedure. Patients classified as “other single right ventricular anomalies” had a heterogenous group of diagnoses with systemic outflow obstruction (e.g. right ventricular dominant atrioventricular septal defect, critical aortic stenosis, straddling mitral valve with left ventricular hypoplasia, etc.) and in some cases anatomy was not delineated in the public use dataset. In addition, the presence of heterotaxy syndrome was not completely presented in the dataset. Patients were excluded if the preoperative cardiac anatomy rendered either the modified Blalock-Taussig shunt (MBTS) or right ventricle-pulmonary artery shunt (RVPAS) technically impossible or if they had any major congenital or acquired extracardiac abnormality that could independently decrease the likelihood of transplant-free survival at 1 year of age. Subjects were randomly assigned to receive either the MBTS or the RVPAS. The institutional review board at each center approved the protocol. Written informed consent was obtained from a parent/guardian before randomization.

All patients enrolled in the publically available SVR database were included in the analysis (n = 549). Patients were randomly divided into a derivation (75%) and validation (25%) cohort. Pre-operative risk factors considered for inclusion are listed in Appendix Table 1. Ascending aorta size was measured intraoperatively by the surgeon. During the SVR trial, patients had a preoperative genetics evaluation by a geneticist only if clinicians suspected an abnormality. For the purposes of this analysis, patients who were evaluated by a geneticist and showed no abnormality in clinical feature or karyotype were labeled as “No genetic syndrome after evaluation.” Patients who were evaluated by a geneticist and did show an abnormality in clinical features or a karyotype were labeled as “Yes genetic syndrome after evaluation.” Patients who were not evaluated by a geneticist due to low suspicion were labeled as “Not evaluated for genetic syndrome.” The primary outcome was hospital mortality/need for cardiac transplantation.

Appendix Table 1.

Variables included in univariate analysis

Age at Norwood

Norwood year

Obstructed total anomalous pulmonary venous return

Partial or unobstructed total anomalous pulmonary venous return

Aberrant right subclavian artery

Norwood volume - surgical site

Norwood volume – individual surgeon

APGARS at 1 and 5 minutes

Pre-operative lactate > 2

Pre-operative intubation

Pre-operative catheterization

Pre-operative other surgery

Pre-operative complication

Abnormal genetic evaluation (evidence of a clinical syndrome or abnormal karyotype)

Multiple gestation

Admission age

Pre-operative discharge

Pre-operative Feeds

Weight

Gender

Race

Ethnicity

Birthweight

Gestational age

Prenatal diagnosis

Fetal intervention

Hypoplastic left heart syndrome subtype

Aortic atresia

Tricuspid regurgitation grade

Tricuspid inflow E deceleration time

Tricuspid inflow E velocity

Tricuspid inflow A velocity

Pulmonary vein reversal duration

Atrial septal defect mean gradient

Right ventricular fractional area change

Right ventricular ejection fraction

Right ventricular end-diastolic volume

Cardiac index

Tricuspid valve annulus Z-score

Tricuspid E:e’

Left ventricular presence

Left ventricular Mass

Left ventricular end-diastolic volume

Right ventricular end-diastolic volume

Pulmonary valve regurgitation grade

Tricuspid tissue Doppler e’ velocity

Tricuspid tissue Doppler a’ velocity

Tricuspid tissue Doppler s’ velocity

Tricuspid tissue Doppler isovolumic acceleration

Ascending aorta diameter

RV myocardial performance index

Tricuspid E:A

Statistical Analysis

All pre-Norwood variables present in the SVR database were assessed for association with hospital mortality by using univariable logistic regression. The variables associated with mortality in univariable analysis (p < 0.2) were entered into a multivariable logistic regression analysis. Continuous variables were converted to categorical variables for the purposes of the multivariable logistic model in order to more feasibly use these variables in the development of a risk score. Continuous variables were converted to categorical variables using three routine cutoff values, the value with the greatest association with hospital mortality was included in the multivariable model. For example, ascending aorta size was tested using cutoff values of 2.0 mm, 2.5 mm, and 3.0 mm. Upon regression, 2.5 mm had the strongest relationship to the outcome, therefore it was tested in the multivariable model. Variables with ≥ 20% of missing data were excluded from the multivariable model. Covariates were eliminated from the final model if they were not independently associated with early mortality, defined as a p-value of < 0.05, and their inclusion did not improve the Nagelkerke R² value by 0.03 or more. In order to maintain clinical usefulness, for categorical variables, if one portion of the category was statistically associated with mortality then the entire variable was used (i.e. in surgeon volume, < 5 Norwood procedures per year was statistically significant, therefore all other surgical volume categories were included in the model). The model’s goodness of fit was tested with the Hosmer-Lemeshow test and Spigelhalter's Z-test to evaluate model calibration, and, receiver operating area under the curve (c-statistic) to evaluate discriminatory power.

Remaining covariates associated with mortality were assigned points based on the relative weight of their estimated beta coefficients in the final multivariable logistic regression model in order to derive a formula for the “Norwood mortality risk score.” A score was then calculated for all patients in the derivation and validation group. The association of the score with hospital mortality was assessed using weighted regression analysis, and logistic regression in both the derivation and validation cohort. Weighted regression was used to account for the unequal distribution of patients between risk scores. In weighted regression analysis, correlations between model predicted mortality rates and actual mortality rates within each respective mortality score were assessed and weights were given based upon the number of patients in each respective mortality score in a 1:1 fashion. Norwood mortality scores were subsequently classified as low, medium and high, with predicted and actual rates of mortality between the three groups compared. Differences between characteristics of the derivation and validation cohorts was assessed using independent t-tests or Chi-square tests. A p-value of < 0.05 was considered statistically significant. All statistics was performed using IBM SPSS® v.22 (New York, NY) and R statistical package (v.3.2.3).

Results

Mortality during the Norwood hospitalization for the entire cohort was 16% (88/549). Comparisons between the derivation and validation cohorts are outlined in Table 1. The cohorts did not differ in sex or age at Norwood. Mortality was similar between groups. Ascending aorta size was not available for 15 patients and was excluded in these patients in a pair-wise fashion.

Table 1.

Comparison between derivation and validation cohorts

	Derivation Cohort (n = 412)	Validation Cohort (n=137)	p-value
Hospital Mortality, n (%)	66 (16%)	24 (18%)	0.68
Age at Norwood (days)	6.7 ± 3.9	7.0 ± 4.4	0.49
Male, n (%)	265 (64%)	75 (55%)	0.06
Low birthweight, n (%)			0.99
< 2500 g	57 (14%)	19 (14%)
≥ 2500 g	355 (86%)	118 (86%)
Clinical syndrome or abnormal chromosomes, n (%)			0.82
Yes after evaluation	20 (5%)	5 (4%)
No after evaluation	236 (57%)	82 (60%)
Not evaluated	156 (38%)	50 (36%)
Surgeon Norwood volume, cases/y (%)			0.83
≤ 5	79 (19%)	29 (21%)
6 to 10	85 (21%)	28 (20%)
11 to 15	178 (43%)	61 (45%)
> 15	70 (17%)	19 (14%)
Anatomic Subtype, n (%)			0.35
HLHS	359 (87%)	115 (84%)
Other	53 (13%)	22 (16%)
Ascending aorta size, n (%)			0.30
≤ 2.5 mm	171 (42%)	52 (38%)
> 2.5 mm	226 (58%)	85 (62%)
Obstructed pulmonary venous return, n (%)			0.50
Yes	13 (3%)	6 (4%)
No	399 (97%)	131 (96%)

HLHS = hypoplastic left heart syndrome

Score derivation

Out of the 52 pre-Norwood variables investigated in the derivation cohort, 24 were considered for inclusion into the multivariable model (p < 0.20). The final multivariable logistic regression model only included covariates that were independently associated with hospital mortality. All variables that were independently associated with mortality improved the model significantly. No variable not independently associated with hospital mortality (p > 0.05) improved the explanatory power of the model significantly. Therefore, all variables not independently associated with the primary outcome were removed from the model. The reason for covariate removal from the final multivariable model is outlined in Table 2.

Table 2.

Covariates with univariable association (p < 0.20) with hospital mortality after the Norwood operation

Included in final model	Excluded from final model: p > 0.05	Excluded from final model: > 20% missing data
Obstructed TAPVC	Age at Norwood	Right ventricular end-diastolic volume
Clinical syndrome or abnormal chromosomes	Gender	Tricuspid Doppler E velocity
Individual surgeon Norwood volume/year	PAPVC or unobstructed3TAPVC	Tricuspid Doppler A velocity
Birth weight	Other associated cardiac diagnoses	Pulmonary vein flow reversal duration
HLHS vs. other anatomy	Institution Norwood volume/year	ASD mean gradient
Ascending aorta diameter	Pre-operative catheterization	Cardiac index
	Pre-operative complication	Tricuspid E:e’
	Multiple gestation
	Gestational Age
	Tricuspid regurgitation severity
	Tricuspid annulus Z-score
	Socioeconomic Score
	Shunt type

ASD = Atrial septal defect, HLHS = Hypoplastic left heart syndrome, PAPVC = Partial anomalous pulmonary venous return, TAPV = Total anomalous pulmonary venous return

The final multivariable logistic regression model consisted of six variables: birthweight, clinical syndrome or abnormal karyotype, surgeon Norwood volume per year, anatomic subtype, ascending aorta size, and obstructed pulmonary venous return. The c-index of the model was 0.79. The Hosmer-Lemeshow test had a p-value of 0.31 and the Spiegelhalter's Z-test had a p-value of 0.07, indicating that the final model fit the data appropriately. The conversion of ascending aorta size and birthweight from continuous variables to categorical variables did not change the explanatory power of the model (Nagelkerke R² = 0.27, p < 0.01 with continuous variables vs. Nagelkerke R² = 0.28, p < 0.01 with categorical variables). To further test the model, each of the six variables used in the final model were used individually to create mortality scores as well as in combinations (i.e. birthweight, then birthweight + ascending aorta size, etc). The performance of scores made from individual variables or combinations of variables was then assessed by c-statistic. No individual variable or combination had c-statistic greater than 0.68, indicating that the final model with all six variables possessed the greatest association with mortality. Whole number scores were assigned to each category based upon the relative magnitude of each individual category’s beta coefficient – the beta coefficient was divided by 0.4 and rounded to the nearest integer. The odds-ratios and resultant scores of the six covariates are shown in Table 3.

Table 3.

Odds ratios and resultant scores after final multivariable model

	Odds Ratio (95% CI)	p-value	Score
Low birthweight, n
< 2500 g	2.90 (1.36 – 6.19)	< 0.01	3
≥ 2500 g	Reference		0
Genetic Syndrome (n)
Not evaluated	5.47 (2.76 – 10.82)	< 0.01	4
Yes after evaluation	3.82 (1.01 – 14.47)	0.05	3
No after evaluation	Reference		0
Individual Surgeon Norwood Volume (cases/y)	6.74 (2.34 – 19.45)	< 0.01	5
≤5	2.28 (0.80 – 6.48)	0.12	2
6 to 10	1.53 (0.57 – 4.11)	0.40	1
11 to 15	Reference		0
> 15
Anatomic Subtype (n)
Other	2.65 (1.11 – 6.34)	0.03	2
HLHS	Reference		0
Ascending aorta size (mm)
≤ 2.5	2.57 (1.32 – 5.01)	< 0.01	2
> 2.5	Reference		0
Obstructed pulmonary venous return (n)
Yes	7.80 (1.39 – 43.75)	0.02	6
No	Reference		0

HLHS = hypoplastic left heart syndrome

Mortality score (range 0–20) was calculated for all patients in the derivation cohort. Using regression analysis weighted for frequency of mortality within an individual score, there was a close association between model predicted rates of mortality and actual rates of mortality (R² = 0.87, p < 0.01) (Figure 1). Upon logistic regression analysis, an increase in score was associated with higher rates of mortality in the derivation cohort, odds ratio 1.48 (95% CI: 1.33 – 1.65, p < 0.01).

Figure 1.

Weighted calibration plots demonstrating the association between predicted mortality rates based on the derivation cohort and observed mortality rates for each risk score in the derivation cohort (top) and validation cohort (bottom). The dashed line represents perfect calibration. The sizes of the bubbles demonstrate the relative weights assigned based on the number of patients with that particular mortality risk score.

Score Validation

Mortality score (range 0–20) was calculated for all patients in the validation cohort. There was no difference between scores in the derivation cohort vs. validation cohort, 5.2 ± 3.2 vs. 5.6 ± 3.5, p = 0.35, respectively.

Using the model developed in the derivation group, predicted rates of mortality were calculated and compared to actual rates of mortality in the validation group. Predicted rates of mortality and actual rates of mortality for each score showed close association in weighted regression analysis (R² = 0.82, p < 0.01) (Figure 1). Upon logistic regression analysis of validation cohort, increasing score was associated with increased chance of hospital mortality, odd ratio 1.39 (95% CI: 1.19 – 1.63, p < 0.01).

To further increase the clinical utility of the mortality score, disjoint categories of low, medium and high risk were determined. Low risk was defined as a score between 0–5, medium risk scores were 6–10, and high risk scores were 11–20. In the derivation cohort, 225 patients (56.7%) were classified as low risk; 150 (37.8%) as medium risk, and 22 (5.5%) high risk. In the validation cohort, 75 (55.6%) had a low risk score, 43 (31.9%) had a medium risk score and 17 (12.6%) had a high risk score. Actual risk of mortality based on score group are displayed in Figure 2. Actual risk of mortality and model predicted risk of mortality were similar for each risk category in both the derivation (R² = 0.99, p < 0.01) and validation cohort (R² = 0.84, p < 0.01). Mortality differed significantly between low, medium, and high risk groups in the derivation (6% vs. 22% vs. 77%, p < 0.01) and validation (4% vs. 30% vs. 53%, p < 0.01) cohorts.

Figure 2.

Mortality Risk by Score Group

Effect of shunt type on mortality within risk categories

In the entire cohort, there was no difference in hospital mortality between patients who received a MBTS vs. a RVPAS in the low risk group (6.5% vs 6.5%, p = 0.99), medium risk group (28.4% vs. 21.7%, p = 0.33), or the high risk group (69.6% vs. 53.3%, p = 0.31). There was no interaction between shunt type and absolute risk score (p = 0.68) or risk score category (p = 0.31).

Discussion

While early mortality rates after the Norwood operation are improving, they remain relatively high.^{2–4, 7–13, 17, 18} While pre-operative patient-related and anatomic risk factors have been previously studied,^{2, 4, 7, 10} they are often grouped with operative and post-operative factors in previous analyses of this dataset,¹⁵ making it difficult to quantify pre-operative risk of hospital mortality after the Norwood procedure. In fact, there is no clinically useful risk stratification tool for neonates with systemic outflow tract obstruction requiring a Norwood operation. Using over 500 neonates in the modern era, we developed and validated a clinically feasible 20-point Norwood mortality risk score and found that patients with a low risk score displayed ~5% risk of hospital mortality, a medium risk score was associated with a ~20–30% risk of hospital mortality, and those with a high risk score showed a > 50% risk of hospital mortality after the Norwood operation.

The components of this clinical score are consistent with previous studies investigating risk factors for mortality after the Norwood procedure. For example, many have identified low birth weight and presence of genetic or non-cardiac abnormalities as risk factors for mortality. ^{2–4, 7, 10} These two risk factors were identified as independent risk factors for mortality even when operative and post-operative risk factor were included in multivariable regression in a previous analysis of this dataset.¹⁵ Smaller ascending aorta size and obstructed total anomalous pulmonary venous return have also been previously been recognized as risk factors for mortality.^{2, 21} We suspect patients undergoing the Norwood operation not classified as hypoplastic left heart syndrome had elevated mortality risk because this group included those with heterotaxy syndrome, which has been shown previously to be a risk factor for mortality.¹⁰ Surgeon Norwood volume per year was a significant risk factor for mortality in the current study. Other studies have shown that center Norwood volume to be an independent risk factor for mortality. ^{2, 17, 18, 22} These findings are likely related, as surgeon and center volume are inherently linked. It is feasible that when only pre-operative factors are analyzed, surgeon volume is independently associated with mortality, but when operative and post-operative factors are accounted for, as in previous studies, center volume accounts for center related factors such as non-surgeon personnel experience, infrastructure factors, etc. that may contribute to mortality risk. However, this study was not designed to assess the relative importance of center volume vs. surgeon volume on hospital mortality in these patients and such assessments merit further study.

During the SVR trial, patients had a preoperative evaluation by a geneticist if clinicians suspected the patient had a genetic syndrome. The need for genetics evaluation was left up to the attending physicians’ discretion. In this cohort, patients who were evaluated and showed no abnormality had lowest risk for mortality. Patients who had an evaluation and showed clinical evidence of a genetic syndrome or an abnormal karyotype had a higher risk of mortality, while those who were not evaluated displayed the highest rate of mortality. It is almost certain that lack of an evaluation by a geneticist did not predispose this cohort to higher hospital mortality after the Norwood operation. Therefore, we can only speculate as to the degree that unaccounted confounders influenced mortality in the group who were not evaluated prior to surgery. It is possible a significant proportion of those not evaluated indeed did have a genetic/syndromic abnormality that was not detected prior to surgery that contributed to a higher rate of mortality. Alternatively, center specific factors in those centers that tended to choose not to perform a genetics evaluation may have confounded the risk of mortality. Unfortunately, individual centers were not identified on the public-use database, and therefore individual center practices could not be accounted for in the analysis. While the “genetic syndrome not evaluated” category is certainly a weakness of the score, this data suggests it may be useful to perform a karyotype and examination by a geneticist prior to the Norwood operation if optimal risk stratification is needed.

The developed Norwood mortality risk score has potential uses in the clinical arena. Cardiologists and surgeons may use the score to counsel parents on the patient-specific risk of performing the Norwood procedure on their child. Hence, it may guide decisions such as surgery vs. comfort care in high risk patients. In addition, some centers may choose a different intervention in the highest risk populations, such as patent ductus arteriosus stenting with pulmonary artery banding. Parents may use the score to guide their decision on where to have surgery. For example, a patient undergoing surgery by a surgeon with low Norwood surgical volume may be in the medium risk category. The same patient could alternatively be in the low risk category if their surgeon performed a higher number of Norwood operations per year.

Another possible use of the score is to increase the accuracy of assessing outcomes after the Norwood procedure. Currently, outcomes after the Norwood procedure are an important criteria used to grade surgeon and center performance. It is feasible that centers who take on more difficult cases exhibit worse outcomes even if the surgeons/centers perform adequately. The most common benchmark for surgeon and center performance in the United States is the Society of Thoracic Surgeon (STS) database. While it does provide risk adjustment for outcomes, components of this score that are currently not used in the STS database may be considered for inclusion to better index such outcomes to case difficulty in order to put center outcomes in a more accurate context.

Limitations

As study participation was limited to 15 participating centers performing 5 or more Norwood procedures annually, the inferences cannot be generalized to centers with smaller case volumes. This data cannot be applied to patients with other major congenital malformation (ex. congenital diaphragmatic hernia) as they were excluded from participation in the SVR trial. There were many variables that were associated with hospital mortality in univariable analysis that had too much missing data to be including in the multivariable analysis. It is possible that these variables may be independently associated with mortality. A number of confounders which may have influenced the results of this study could not be accounted for in this study as they were not reported in the dataset (such as the presence of heterotaxy syndrome). Therefore, our results should be externally validated. Our results are very similar with Tweddell et al. and Tabbutt et al. who both investigated the relationship between pre-operative risk factors and hazard of early death.^{15, 16} There were a number of pre-operative factors found in these studies that were not included in the current analysis, such as the interaction between shunt type and preterm status and aortic valve patency as these are not clinically feasible to assess when calculating a score. While there is some overlap between our studies, the previous analysis was not restricted to hospital mortality and included shunt-type (an operative factor) as a co-factor, therefore, we felt a separate analysis was appropriate to reach our aims in order to develop a clinically feasible Norwood hospital mortality risk score.

Conclusion

A 20 point early Norwood hospital mortality risk score in neonates with systemic outflow obstruction was successfully derived and validated. The risk score accurately distinguishes patients at high, medium and low risk for hospital mortality using easily obtainable clinical factors. The score has the potential to be used in clinical practice to aid in risk assessment prior to surgery.

Central Message.

A hospital mortality risk score was validated in neonates undergoing the Norwood operation to aid in pre-operative risk assessment.

Perspective Statement.

The ability to quantify patient-specific hospital mortality risk prior to the Norwood procedure remains elusive. We developed and validated an accurate and clinically feasible score to assess the risk of hospital mortality in neonates undergoing the Norwood procedure. The score has the potential to be used in clinical practice to aid in risk assessment prior to surgery.

Abbreviations

HLHS

hypoplastic left heart syndrome

MBTS

modified Blalock-Taussig shunt

RVPAS

right ventricle-pulmonary artery shunt

STS

Society of Thoracic Surgeons

SVR

single ventricle reconstruction