Population-based Validation of a Clinical Prediction Model for Congenital Diaphragmatic Hernias

Abstract

Objective

To examine the external validity of a well-known CDH clinical prediction model using a population-based cohort.

Study design

Newborns with CDH born in California between 2007–2012 were extracted from the Vital Statistics and Patient Discharge Data (VS-PDD) Linked Files. The total CDH risk score was calculated according to the Congenital Diaphragmatic Hernia Study Group (CDHSG) model using five independent predictors: birth weight, 5-minute Apgar, pulmonary hypertension, major cardiac defects, and chromosomal anomalies. CDHSG model performance on our cohort was validated for discrimination and calibration.

Results

705 CDH newborns were extracted from 3,213,822 live births. CDH newborns were delivered in 150 different hospitals, whereas only 28 hospitals performed CDH repairs (1–85 repairs per hospital). The observed mortality for low-, intermediate-, and high-risk groups were 7.7%, 34.3%, and 54.7%, and predicted mortality for these groups were 4.0%, 23.2%, and 58.5%. The CDHSG model performed well within our cohort with a C-statistic of 0.741 and good calibration.

Conclusion

We successfully validated the CDHSG prediction model using an external population-based cohort of CDH newborns in California. This cohort may be used to investigate hospital volume-outcome relationships and guide policy development.

Congenital diaphragmatic hernia (CDH) has a reported incidence of 1.76–2.3 per 10,000 live-births (1–3). Despite the low incidence, the projected burden of CDH patients may exceed $250 million annually in the United States, especially if patients require aggressive treatment such as extracorporeal membrane oxygenation (4). Along with these high costs, there is a large, burden of mortality that has only decreased from about 34% to 29% in the last 2 decades (5).

There have been a number of clinical prediction models attempting to risk stratify CDH infants based on their presentation (6–10). Institutions and hospital networks have used these models to demonstrate systematic improvement in the outcomes of CDH (11–14). A standardized risk adjustment tool could better compare therapeutic strategies and outcomes by stratifying patients across institutions and hospital systems.

In 2014, Brindle et al and the Congenital Diaphragmatic Hernia Study Group (CDHSG) published an updated, simplified clinical prediction rule to stratify CDH infants based on disease severity. This prediction model used the CDHSG registry to develop a total CDH risk score from clinical indicators at birth based on birth weight, 5-minute Apgar score, severe pulmonary hypertension, and the presence of major cardiac or congenital anomalies. The model can be easily applied within the first few hours of life to accurately stratify CDH newborns into low-, intermediate-, and high-risk of mortality groups (15), and has been shown to have good discrimination between the risk groups in a single institution’s cohort (16). However, the CDHSG model has not been externally validated at the population level. External validation of clinical prediction models uses novel participant level data to examine whether the model’s predictions are reliable in individuals for clinical use (17). The aim of our study was to externally examine the validity of the CDHSG clinical prediction model on a non-voluntary, statewide cohort of newborns with CDH.

Methods

We used the Vital Statistics and Patient Discharge Data (VS-PDD) Linked Files from the California Office of Statewide Health Planning and Development (OSHPD) for our data. This dataset links the Vital Statistic birth records to maternal and infant hospital inpatient discharges from all non-federal hospitals in the State of California. Each record captures a single hospital encounter and includes birth vital statistics, patient demographics, facility identification, dates of services, and codes for diagnoses and procedures during that encounter. All diagnosis and procedural codes are categorized according to the International Statistical Classification of Disease, Ninth Revision (ICD-9). Neonatal and maternal information from the VS-PDD are linked through a probabilistic matching algorithm internal to the OSHPD and produces a unique birth identification number that allows us to identify all hospital encounters for a particular infant in the state of California before one year of age. This database has been used previously to examine health outcomes in neonates (18,19).

Neonates with CDH in California born between January 1, 2007 and December 31, 2012 were extracted from the VS-PDD Linked Files. The criteria for inclusion was a newborn with a birth record containing an ICD-9 diagnosis code “Congenital Anomalies of Diaphragm” (756.6) (20). Only newborns with a de-identified birth identification code were included and these infants were tracked until in-hospital death or discharge from the acute care setting. CDH surgical repair was identified using ICD-9 procedure codes 53.70–53.84, and excluded “Plication of the diaphragm” (53.81) and “Repair of parasternal hernia” (53.82). CDH was considered lethal if not repaired; therefore, newborns with ICD-9 code of 756.6 and discharged alive without surgical repair were considered to have diaphragmatic eventration and were excluded. A similar method has previously been used to identify CDH infants based on ICD-9 codes (21). Infants with incomplete birth information necessary for the CDHSG prediction model were also excluded.

CDH newborn encounters were reviewed for clinical data including gestational age, birth weight, and 5-minute Apgar score. Severe pulmonary hypertension was identified at the birth hospital encounter and defined by the ICD-9 code for “Persistent Pulmonary Hypertension” (747.83). Major cardiac anomalies were identified by ICD-9 code diagnoses and defined as all anomalies excluding patent atrial septal defects or patent ductus arteriosus (15,22). Chromosomal anomalies were defined as any ICD-9 diagnostic code 758.0 – 758.9. Binary independent predictors were created, as defined in the CDHSG model, for low birth weight (<1500g), low 5-minute Apgar score (<7), missing 5-minute Apgar score, and the presence of pulmonary hypertension at birth, major cardiac defects, and chromosomal anomalies.

The CDHSG clinical prediction model was applied to the binary independent predictors to calculate a total CDH risk score (Table I). The total CDH risk scores (range 0–8) were used to stratify newborns into three risk groups: Low-Risk (0), Intermediate-Risk (1–2), and High-Risk (3–8) (15).

Table 1.

Values for each of the independent predictors to calculate the total CDH risk score according to the 2014 CDH Study Group Clinical Prediction Model.

Model Variable	Value
Low Birth Weight	1
Low 5-min Apgar	1
Missing 5-min Apgar	2
Pulmonary Hypertension at Birth	2
Major Cardiac Anomaly	2
Chromosomal Anomaly	1
Total CDH Risk Score	0–8

Statistical Analyses

Newborn characteristics at baseline are described using medians and inter-quartile ranges for continuous variables and percentages for categorical variables. Bivariate comparisons between baseline characteristics and death before discharge were made using the Fisher exact test for categorical variables and nonparametric Wilcox rank sum test for continuous variables. The CDHSG model was used to calculate individual predicted outcome probabilities. The predicted outcome was then compared with the observed outcome in the study cohort. The model performance was assessed for discrimination using the c-statistic (23). The c-statistic ranges between 0.5 (poor discrimination) to 1.0 (perfect discrimination) and represents the probability that among a randomly selected pair of observations with different outcomes, predicted outcome risk is higher in the case with the outcomes of interest (24). The predicted and observed outcomes were also assessed for calibration by plotting a calibration curve, by performing the Hosmer-Lemeshow test, and deriving Harrell E-Statistic. Harrell E-statistic, the mean absolute difference between predicted and smoothed observed values, estimates the degree of variability in the estimation between the observed and predicted values based on a fitted logistic regression (25). All statistical tests were 2-tailed and a P value <.05 was considered statistically significant. Data analysis was performed using R (R Core Team, Vienna, Austria) and STATA/MP 12.1 (StataCorp LP, College Station, TX).

This retrospective population-based cohort study was performed after obtaining institution review board approval from both Tufts Medical Center (Boston, MA) (#11349) and the Committee for Protection of Human Subjects in the State of California (14-09-1714).

Results

There were 3,213,822 live-births in California between January 1, 2007 and December 31, 2012. A total of 753 newborns had an ICD-9 diagnosis of 756.6. Forty-eight infants were excluded due to incomplete birth information or incomplete hospital discharge records leaving a total of 705 newborns with CDH in our study cohort (Figure 1; available at www.jpeds.com). The average incidence of CDH was 2.2 per 10,000 live births or 1 in 4,430 live births. The overall survival rate was 68.7%.

Figure 1.

Inclusion and exclusion criteria for patient selection.

The majority of newborns in our cohort were male (58.6%). The median gestational age was 38.7 weeks and median birth weight was 3000 grams. The neonates had a median 5-minute Apgar score of 8. Pulmonary hypertension or major cardiac anomalies were present in 212 (30.1%) and 130 (18.4%) infants respectively. Overall 56 (7.9%) of infants had a chromosomal anomaly. The most common major congenital cardiac defects were ventricular septal defects in 77 (10.9%), persistent truncus arteriosus in 37 (5.3%), and coarctation of aorta in 29 (4.11%). Baseline characteristics of CDH survivors were significantly different compared with those who did not survive for the model parameters of low birth weight, low 5-minute Apgar score, major cardiac defect, and chromosomal anomaly. There were no significant differences with respect to missing 5-minute Apgar and pulmonary hypertension (Table 2).

Table 2.

Baseline characteristics of newborns with CDH according to survival (N = 705).

Baseline Characteristics	All patients N=705	Non-survivors n = 221	Survivors n = 484	p Value *
Male	413 (58.6)	123 (55.7)	290 (59.9)	0.323
Median gestational age at birth, wk	38.7 (37.0–39.8)	38.0 (35.7–39.3)	39 (37.3–40.0)	<0.001
Preterm (<37 wk GA)	170 (24.7)	71 (34.1)	99 (20.9)	0.001
Median birth weight, g	3000 (2475–3500)	2700 (1849–3175)	3130 (2680–3556)	<0.001
Low birth weight (<1500 g)	53 (7.5)	35 (15.8)	18 (3.7)	<0.001
Median 5-min Apgar score	8 (5–9)	5 (3–8)	8 (7–9)	<0.001
Low Apgar score (<7)	241 (37.5)	131 (67.2)	110 (24.6)	<0.001
Missing Apgar score	62 (8.8)	26 (11.8)	36 (7.4)	0.064
Pulmonary hypertension at birth	212 (30.1)	74 (33.5)	138 (28.5)	0.185
Major cardiac defect	130 (18.4)	71 (32.1)	59 (12.2)	<0.001
Chromosomal anomaly	56 (7.9)	39 (17.2)	17 (3.5)	<0.001
Overall survival	484 (68.7)

Data are presented as n (%) or median (IQR). GA, gestational age.

^*Fishers exact test performed for all binary variables, Wilcoxon rank sum test for all continuous variables.

The distribution of patients according to total CDH risk score is shown in Figure 2, A. The proportion of low- (33%), intermediate- (38%), and high-risk (29%) groups was balanced (Figure 2, B). The observed in-hospital mortality rate increased progressively from 7.7% for a low-risk patient with a total CDH risk score of zero to 72.7% for a newborn with a total CDH risk score of six. For the intermediate risk group, there was a lower mortality rate of 40.6% and 28.7% between CDH risk scores of 1 and 2 respectively (Table 3; available at www.jpeds.com). This trend of increasing mortality with increasing total CDH risk score was also evidenced after grouping the model scores into the three distinct risk groups (Figure 3). The observed mortality for the low-risk group was 7.7%, and the predicted mortality rate was 4.0%. For the intermediate-risk group, the observed mortality rate was 34.3% and predicted was 23.2%. The high-risk group had observed mortality rate of 54.7% and predicted mortality rate of 58.5%. The CDHSG model’s discrimination in our study cohort was moderately strong with a c-statistic of 0.741. The model also had good calibration shown in Figure 3. The Hosmer-Lemeshow chi square test was significant (chi-square = 23.0, p<0.001). Harrell E statistics demonstrated that the observed-to-expected mortality rate differed by 6.6% on average for this cohort.

Figure 2.

(A) Number of CDH infants for each total CDH risk score. (B) Distribution of infants into three risk groups. Low-, intermediate-, and high-risk CDH groups have total CDH risk score of 0, 1–2, and 3–6 respectively.

Table 3.

Distribution of survival by total CDH risk score.

	Total CDH Risk Score
	0	1	2	3	4	5	6
Died (%)	18 (7.7)	52 (40.6)	41 (28.7)	56 (54.9)	33 (48.5)	13 (65.0)	8 (72.7)
Lived (%)	215 (92.3)	76 (59.4)	102 (71.3)	46 (45.1)	35 (51.5)	7 (35.0)	3 (27.3)

Data are presented as n (%).

Figure 3.

Observed and predicted mortality rates for low-, intermediate-, and high-risk CDH groups according to total CDH risk score (0–6).

The 705 CDH newborns in our study cohort were delivered at 150 unique hospitals throughout the state of California. Overall, 48.9% of the newborns were delivered at 124 hospitals that did not perform any CDH repairs during the study period. There were 28 hospitals that performed CDH repairs (range 1–85 repairs per hospital). Fifty-two newborns (7.4%) died in a hospital without capacity to perform CDH repair prior to being transferred to a definitive care hospital. Of these 52 newborns, there were 2 infants in the low-risk group who both died on day of life 1, 29 newborns in the intermediate risk group of which 96.6% died by day of life 2, and 21 newborns in the high-risk group of which 85.7% died by day of life 2. Furthermore, subgroup analysis examining the lower mortality rate within the intermediate risk group showed 79 out of 128 (61.7%) infants and 52 out of 143 (36.3%) infants with CDH risk score of 1 and 2 respectively were born at hospitals that did not perform any CDH repairs during the study period.

Discussion

Model development, external validation and impact evaluation are the three fundamental components of prediction model research (26). Most model prediction research focuses on model development and there is little evidence about which models are reliable and under what circumstances (17). Model development has led to risk-assessment tools for risk stratifying CDH newborns based on birth weight and Apgar scores, Score for Neonatal Acute Physiology version II (SNAP-II), and Wilford Hall/Santa Rosa (WHSR_PF) score derived from blood gases (6,7,9). External validation of these models in a large, diverse population with assessment of calibration and discrimination is essential to examining model utility and has been performed previously on the CDHSG, SNAP-II, and WHSR_PF models (8).

Our study provides population-based external validation of the 2014 clinical prediction model for the severity of CDH published by the CDHSG registry. We found that the California VS-PDD cohort of 705 CDH neonates could be accurately risk stratified into low-, intermediate-, and high-risk groups using the CDHSG clinical prediction rule. The CDHSG model c-statistic of 0.741 for our cohort compared favorably with the c-statistic of 0.769 in the validation model of the original study (15). The observed mortality increased with increasing total CDH risk scores in close proximity with the predicted mortality rates from the model and there was no evidence of miscalibration on the calibration plot. Harrell E-statistic indicated that the model may slightly over-estimate mortality for each risk group by the magnitude of 6.6%. However, this slight over-estimation is likely to be clinically insignificant as there is still a very distinct, significant difference in mortality between the three risk groups and clinical management decisions are unlikely to change due to this difference. Overall, the CDHSG model had moderately strong discrimination in our study cohort and had good calibration as seen on both the calibration plot and Harrell E-statistic.

The current study also covers CDH care across the state of California, including the entire spectrum from academic, freestanding children’s hospitals to lower-volume community hospitals. The use of population-based data for our study avoids inherent selection bias of a voluntary disease-specific registry in selected high-volume tertiary children’s hospitals (27). Our study was able to capture the outcomes of CDH patients that may not be identified through databases that consist primarily of larger tertiary hospitals, such as the CDHSG, Pediatric Health Information System (PHIS), Diaphragmatic Hernia Research and Exploration, Advancing Molecular Science (DHREAMS) or larger single institutions (14,28–30). Given the broad spectrum of hospitals, it is interesting to note that in analyzing the non-linear mortality rate between intermediate CDH risk scores of 1 and 2, there is a greater proportion of infants with a CDH risk score of 1 who were born at hospitals that did not have the capacity to perform CDH repair. Thus within the intermediate CDH risk group, the lack of CDH repair capabilities at a hospital where a patient was born may help explain the higher mortality for CDH risk score 1 relative to 2. By including infants at all hospitals, we provide a more accurate representation that avoids the ‘hidden mortality’ for institutionally reported CDH survival rates (31).

Our study identified an incidence of 2.2 CDH cases per 10,000 live births, which is similar to those reported by epidemiological studies ranging from 1.76–2.3 per 10,000 live births. We found a survival rate of 68.7%, which was also comparable with epidemiological studies (54.11%–69.3%) (1–3). The baseline characteristics for cardiac and other anomalies in our California CDH cohort was also similar to previous studies (10,22,28,30,32). These findings support our conclusion that the California VS-PDD cohort provides an accurate depiction of the population of infants with this congenital anomaly.

There are a number of limitations to this study. First, the VS-PDD Linked Files do not contain certain clinical variables, such as laboratory and radiologic findings, but the database contains all the information required by the CDHSG model to accurately risk stratify the CDH neonates. In particular, the lack of echocardiographic timing and measurements may have underestimated the true incidence of persistent pulmonary hypertension and the correlation between persistent pulmonary hypertension and clinical outcomes in this cohort. We based the presence of pulmonary hypertension of the study population from discharge diagnoses rather than the first echocardiogram findings as detailed in Brindle et al (15). However, despite the differences in the way pulmonary hypertension was classified, the CDHSG clinical prediction model still had robust discrimination and calibration within our study population. Second, there may be variability and inaccuracies between hospitals in recording data, however the OSHPD internally validates the database through multiple check points leading to an error tolerance level of less than 2%. Finally, we relied on the group estimations of the CDHSG prediction model output rather than making individual predictions because the original CDH study model lacked a model intercept to enable individual risk predictions (Brindle, personal communication). Despite the grouped prediction, the CDHSG model still performed well in our study cohort and the model accurately risk stratified a non-voluntary, statewide cohort of CDH neonates. Our study linked databases for different healthcare settings and identified individual CDH newborn’s birth vital statistics, initial birth records and subsequent hospital transfers within the state of California. In addition, using population-based data from a large, diverse state population constituting 12% of the US population, we identified robust estimates on the factors that affect CDH newborn survival.

In conclusion, we have externally validated the 2014 CDHSG clinical prediction model using a population-based cohort. The population-based validation of the CDHSG model strengthens the applicability of the total CDH risk score to accurately predict CDH outcomes across different healthcare systems. The risk stratified population-based cohort of CDH newborns in our study may be used in the future to identify ideal resource allocation based on disease severity and guide policy development in the United States.

Abbreviations

CDH

Congenital diaphragmatic hernia

VS-PDD

Vital Statistics and Patient Discharge Data

CDHSG

Congenital Diaphragmatic Hernia Study Group

OSHPD

Office of Statewide Health Planning and Development

ICD-9

International Statistical Classification of Disease, Ninth Revision