Skip to main content
JAMA Network logoLink to JAMA Network
. 2018 Sep 10;172(11):1061–1069. doi: 10.1001/jamapediatrics.2018.2029

Association of Race/Ethnicity With Very Preterm Neonatal Morbidities

Teresa Janevic 1,2,3,, Jennifer Zeitlin 3,4, Nathalie Auger 5, Natalia N Egorova 3, Paul Hebert 6, Amy Balbierz 1,3, Elizabeth A Howell 1,2,3
PMCID: PMC6248139  PMID: 30208467

Key Points

Question

Are there racial/ethnic disparities in very preterm neonatal morbidity?

Findings

In this population-based cohort study of 582 297 very preterm birth infants, substantial disparities in necrotizing enterocolitis, intraventricular hemorrhage, bronchopulmonary dysplasia, and retinopathy of prematurity were found between black and white infants as well as between Hispanic and white infants, and disparities in retinopathy of prematurity were also notable between Asian and white infants. Disparities measured instead with a standard approach in a very preterm birth cohort were attenuated or null.

Meaning

Racial/ethnic disparities in very preterm neonatal morbidity are sizeable, and reports from very preterm birth cohorts may underestimate the magnitude of these disparities.

Abstract

Importance

Severe morbidity in very preterm infants is associated with profound clinical implications on development and life-course health. However, studies of racial/ethnic disparities in severe neonatal morbidities are scant and suggest that these disparities are modest or null, which may be an underestimation resulting from the analytic approach used.

Objective

To estimate racial/ethnic differences in severe morbidities among very preterm infants.

Design, Setting, and Participants

This population-based retrospective cohort study was conducted in New York City, New York, using linked birth certificate, mortality data, and hospital discharge data from January 1, 2010, through December 31, 2014. Infants born before 24 weeks’ gestation, with congenital anomalies, and with missing data were excluded. Racial/ethnic disparities in very preterm birth morbidities were estimated through 2 approaches, conventional analysis and fetuses-at-risk analysis. The conventional analysis used log-binomial regression to estimate the relative risk of 4 severe neonatal morbidities for the racial/ethnic groups. For the fetuses-at-risk analysis, Cox proportional hazards regression with death as competing risk was used to estimate subhazard ratios associating race/ethnicity with each outcome. Estimates were adjusted for sociodemographic factors and maternal morbidities. Data were analyzed from September 5, 2017, to May 21, 2018.

Main Outcomes and Measures

Four morbidity outcomes were defined using International Classification of Diseases, Ninth Revision, diagnosis and procedure codes: necrotizing enterocolitis, intraventricular hemorrhage, bronchopulmonary dysplasia, and retinopathy of prematurity.

Results

In total, 582 297 infants were included in this study. Of these infants, 285 006 were female (48.9%) and 297 291 were male (51.0%). Using the conventional approach in the very preterm birth subcohort, black compared with white infants had an increased risk of only bronchopulmonary dysplasia (adjusted risk ratio [aRR], 1.34; 95% CI, 1.09-1.64) and a borderline increased risk of necrotizing enterocolitis (aRR, 1.39; 95% CI, 1.00-1.93). Hispanic infants had a borderline increased risk of necrotizing enterocolitis (aRR, 1.39; 95% CI, 0.98-1.96), and Asian infants had an increased risk of retinopathy of prematurity (aRR, 1.85; 95% CI, 1.15-2.97). In the fetuses-at-risk analysis, black infants had a 4.40 times higher rate of necrotizing enterocolitis (95% CI, 2.98-6.51), a 2.73 times higher rate of intraventricular hemorrhage (95% CI, 1.63-4.57), a 4.43 times higher rate of bronchopulmonary dysplasia (95% CI, 2.88-6.81), and a 2.98 times higher rate of retinopathy of prematurity (95% CI, 2.01-4.40). Hispanic infants had an approximately 2 times higher rate for all outcomes, and Asian infants had increased risk only for retinopathy of prematurity (adjusted hazard ratio, 2.43; 95% CI, 1.43-4.11).

Conclusions and Relevance

In this study, racial/ethnic disparities in neonatal morbidities among very preterm infants appear to be sizable, but may have been underestimated in previous studies, and may have implications for the future. Understanding these racial/ethnic disparities is important, as they may contribute to inequalities in health and development later in the child’s life.


This cohort study uses birth certificate, mortality, and hospital discharge data to investigate whether race/ethnicity variables increase the risks of severe morbidities among very preterm newborns.

Introduction

Racial/ethnic disparity in neonatal mortality associated with the increased risk of very preterm birth among black infants is well documented.1 However, racial/ethnic disparities in severe neonatal morbidities among very preterm infants are less studied, despite the association of these morbidities with life-course health disparities.2 In the past few decades, infant survival very early in gestation has increased steadily because of advances in medical technology and quality improvement initiatives.3 However, very preterm infants often face severe respiratory, digestive, neurologic, and neurosensory morbidity because of immaturity.4 Understanding racial/ethnic differences in severe morbidity is important because these disparities may contribute to inequalities in health and development in later life.

Research on racial/ethnic disparities in severe morbidities among preterm infants has drawn varied conclusions. Some studies show an increased risk for some severe morbidities for black and Hispanic infants, whereas others report similar or lower risk of morbidities for this population.5,6,7,8 Necrotizing enterocolitis (NEC), an inflammatory disease of the intestine that is a leading cause of death among preterm infants,9 is an example of an important neonatal morbidity for which findings on black-white disparities are small or absent.5,6,7,8,10,11 Despite plausible mechanisms for black-white disparities in NEC (including antenatal factors, such as psychosocial stress12 and maternal cigarette smoking,13 and postnatal factors, such as quality of care14,15 and human-milk feeding16) reported disparities are minimal. One explanation for the null or varied findings on disparities in NEC and other neonatal morbidities may lie in the method used by researchers who study cohorts selected by gestational age.

Evidence is accumulating that measures of association for factors present before birth can be biased when infants are stratified on birth weight or gestational age.17 Given that the cause of preterm birth is associated with various pathophysiologic processes,18 infants in preterm cohorts are selected on the basis of other factors that may be associated with the prenatal characteristic of interest—in this case, race/ethnicity.19 This phenomenon, known as the birth-weight paradox, was first observed in infants who were born to smokers but had a survival advantage at very low birth weights or gestational ages20; the phenomenon was later observed among black infants.21 Although at first attributed to physiologic differences, the paradoxical associations found at very low birth weight or early gestational age are likely from comparing groups of infants whose preterm births are associated with different unmeasured causes that are also associated with the maternal risk factor of interest.22 The selection of high-risk white infants into the very preterm birth cohort, then, can result in null or paradoxically protective association between black race and neonatal morbidity. The clinical implications of such biased associations are profound and can lead to conclusions that might affect practice, such as black women have a shorter normal length of gestation.23

One method proposed for addressing biased paradoxical associations, commonly called collider stratification bias,24 is the fetuses-at-risk (FAR) approach. With FAR, the denominator for an outcome at a given gestational age includes both delivered and ongoing (not delivered) pregnancies.25,26,27,28 Our objective for this study was to estimate racial/ethnic differences in severe neonatal morbidity among very preterm infants in a diverse, population-based setting, using both a conventional approach in a subcohort restricted to very preterm births and the FAR approach. Using linked birth certificate and hospital discharge data from New York City, New York, from 2010 to 2014, we investigated the associations between race/ethnicity and 4 severe neonatal morbidities: NEC, intraventricular hemorrhage (IVH), retinopathy of prematurity (ROP), and bronchopulmonary dysplasia (BPD).

Methods

Study Sample

We conducted a retrospective cohort study using linked birth certificate, hospital discharge (data from Statewide Planning and Research Cooperative System [SPARCS]), and mortality data from January 1, 2010, through December 31, 2014. As reported previously,14 the New York State Department of Health and Human Services linked these data using a probabilistic linking methodology, and 99.9% of infant discharge abstracts were linked with infant birth certificates. The linkage rate with maternal discharge abstracts was 96%. Institutional review board approvals were obtained from the New York City Department of Health and Mental Hygiene, the New York State Department of Health, and the Icahn School of Medicine at Mount Sinai, in New York City. Informed consent was waived by the Icahn School of Medicine at Mount Sinai. Data were analyzed from September 5, 2017, to May 21, 2018.

The linked data set included birth data, death data, and SPARCS records for all infants born between 2010 and 2014 who were discharged by December 31, 2014, and who died between 2010 and 2015, for a total of 596 295 births. We excluded infants born before 24 weeks’ gestation (n = 774) and infants with congenital anomalies (n = 10 757), whom we identified using diagnosis codes from the infant SPARCS record (eTable 1 in the Supplement).29 We also excluded infants with missing values for gestational age, race, or ethnic ancestry or other race (n = 2029), leaving a total of 582 297 infants (97.7%) in the analytic sample. We subset the data further to create a very preterm birth (VPTB) subcohort of infants born between 24 and 31 weeks’ gestation (n = 7155 [1.2%]).

Neonatal Morbidities

We defined 4 severe neonatal morbidities using International Classification of Diseases, Ninth Revision, diagnosis and procedure codes for NEC (unspecified, stages 2-3, laparotomy), IVH (grades 3-4), BPD, and ROP (stages 3-5) (eTable 2 in the Supplement).30,31

Race/Ethnicity and Covariates

We obtained information on maternal race and ethnic ancestry from the birth certificates and created a race/ethnicity variable by combining race and Hispanic ethnicity into the following categories: non-Hispanic black (henceforth referred to as black, for brevity), Hispanic, non-Hispanic white (white), and Asian. We also obtained from the birth certificates information on maternal sociodemographic characteristics (age, nativity, educational level, and insurance status), maternal behaviors during pregnancy (tobacco, alcohol, and drug use and prenatal care visits), maternal medical risk factors (parity, comorbidities, pregnancy complications, and body mass index), and infant factors (multiple birth, sex, birth weight, and clinical estimate of gestational age). Maternal comorbidities were ascertained using a combination of the mother’s SPARCS record and birth certificate (eTable 3 in the Supplement). To maximize the sensitivity of our measures, we classified morbidity as present if it was indicated either on the SPARCS record or the birth certificate .32

Statistical Analysis

We examined bivariate associations between sample characteristics and race/ethnicity using the χ2 test. To estimate the associations between race/ethnicity and neonatal morbidities, we compared 2 approaches: conventional analysis and FAR analysis. The FAR approach estimates the totality of the direct effect of race/ethnicity on VPTB morbidity as well as the indirect effect by gestational age. In contrast, much of the current literature on VPTB morbidities attempts to estimate the direct effect of race/ethnicity by stratifying for gestational age, which is subject to collider stratification bias.

We conducted the conventional analysis in a subcohort of infants born between 24 and 31 weeks’ gestation. Using log-binomial regression, we calculated the relative risk of each severe neonatal morbidity for black, Hispanic, and Asian infants as compared with white infants using log-binomial regression.33 We created 4 models for each morbidity outcome (NEC, IVH, BPD, and ROP). We estimated the unadjusted association in model 1, and we adjusted a series of covariates in models 2 to 4 to provide a range of estimates, given that most covariates could plausibly be on the causal pathway between race/ethnicity and infant morbidities. Covariates were chosen from previous literature on maternal and infant risk factors for neonatal morbidity. Covariates were categorized as displayed in Table 1. In model 2, we adjusted for sociodemographic characteristics (maternal age, parity, educational level, insurance status, and nativity; infant sex). In model 3, we additionally adjusted for maternal morbidities (pregestational hypertension, gestational hypertension, pregestational diabetes, gestational diabetes, placenta previa, obesity, and smoking status). In model 4, we additionally adjusted for gestational age and birth-weight z score, which is sometimes used to adjust for fetal growth restriction.34

Table 1. Sample Characteristics by Race/Ethnicity, New York City, 2010-2014.

Variable No. (%)
Black Hispanic Asian White
Total 120 842 (100) 176 835 (100) 97 458 (100) 187 162 (100)
Gestational age, wk
24-28 1084 (0.9) 710 (0.4) 209 (0.2) 404 (0.2)
29-31 1147 (1.0) 935 (0.5) 357 (0.4) 672 (0.4)
32-36 12 046 (10.0) 13 666 (7.7) 6629 (6.8) 12200 (6.5)
37-42 106 565 (88.2) 161 524 (91.3) 90 263 (92.6) 173 886 (92.9)
Maternal age, y
<20 8509 (7.0) 132 406 (9.0) 877 (0.9) 2292 (1.2)
20-34 87 602 (72.5) 132 406 (74.9) 73 110 (75.0) 128 659 (68.7)
35-39 18 477 (15.3) 22 358 (12.6) 18 694 (19.2) 42 795 (22.9)
40-44 5688 (4.7) 5722 (3.2) 4407 (4.5) 12 117 (6.5)
≥45 566 (0.5) 366 (0.2) 370 (0.4) 1299 (0.7)
Maternal educational level
Less than high school 24 291 (20.1) 64 733 (36.6) 21 652 (17.3) 14 550 (7.8)
High school 32 408 (26.8) 42 427 (24.0) 17 957 (18.4) 35 033 (18.7)
Some college/trade school 38 616 (32.0) 44 791 (25.3) 15 763 (16.2) 27 655 (14.8)
College degree or higher 24 865 (20.6) 24 464 (13.8) 42 017 (43.1) 109 481 (58.5)
Missing 662(0.6) 420 (0.2) 69 (0.1) 443 (0.2)
Maternal insurance status
Medicaid 84 724 (70.1) 138 639 (78.4) 58 673 (60.2) 63 722 (34.1)
Private 30 562 (25.3) 34 076 (19.3) 37 392 (38.4) 120 961 (64.6)
Self-pay 3332 (2.8) 2443 (1.4) 812 (0.8) 1106 (0.6)
Other 2224 (1.8) 1677 (1.0) 581 (0.6) 1373 (0.7)
Trimester-initiated prenatal care
1st 74 546 (61.7) 116 900 (66.1) 70 909 (72.8) 150 203 (80.3)
2nd 29 021 (24.0) 43 256 (24.5) 19 785 (20.3) 28 582 (15.3)
3rd or none 12 929 (10.7) 12 359 (7.0) 5471 (5.6) 5681 (3.0)
Missing 4346 (3.6) 4320 (2.4) 1293 (1.3) 2696 (1.4)
Infant birth
Singleton 116 208 (96.2) 172 133 (97.3) 94 520 (97.0) 177 567 (94.9)
Multiple 4634 (3.8) 4702 (2.7) 2938 (3.0) 9595 (5.1)
Infant sex
Male 61 218 (50.7) 89 388 (50.6) 50 320 (51.6) 96 365 (51.5)
Maternal BMI
Underweight (<18.5) 4245 (3.5) 5420 (3.1) 11 140 (6.0) 11 477 (11.8)
Normal weight (18.5-24.9) 45 469 (37.6) 79 304 (44.9) 123 609 (66.0) 65 507 (67.2)
Overweight (25.0-29.9) 35 310 (29.2) 52 368 (29.6) 34 266 (18.3) 15 255 (15.7)
Obese (30.0-39.9) 28 805 (23.8) 33 228 (18.8) 15 469 (8.3) 4720 (4.8)
Morbid obesity (≥40) 5699 (4.7) 4570 (2.6) 1886 (1.0) 232 (0.2)
Missing 1314 (1.1) 1945 (1.1) 792 (0.4) 267 (0.3)
Maternal parity
Nulliparous 52 234 (43.2) 72 249 (40.9) 48 838 (50.1) 86 840 (46.4)
Multiparous 68 404 (56.6) 104 400 (59.0) 48 561 (49.8) 100 138 (53.5)
Missing 204 (0.2) 186 (0.1) 59 (0.1) 184 (0.1)
Maternal nativity
US born 69 735 (57.7) 80 637 (45.6) 23 350 (24.0) 135 806 (72.6)
Foreign born 51 107 (42.3) 96 198 (54.4) 74 108 (76.0) 51 356 (27.4)
Smoked during pregnancy 3966 (3.3) 4158 (2.4) 758 (0.8) 4212 (2.3)
Used alcohol during pregnancy 904 (0.8) 1010 (0.6) 596 (0.6) 1409 (0.8)
Used drugs during pregnancy 789 (0.7) 547 (0.3) 30 (0.03) 242 (0.1)
Maternal morbidity
Placental abruption 1358 (1.1) 1645 (0.9) 1716 (0.9) 1157 (1.2)
Pregestational diabetes 1582 (1.3) 1758 (1.0) 705 (0.7) 846 (0.5)
Gestational diabetes 8705 (7.2) 12 772 (7.2) 13 460 (13.8) 8955 (4.8)
Pregestational hypertension 15 795 (13.1) 16 288 (9.2) 4509 (4.6) 10 009 (5.6)
Gestational hypertension/preeclampsia 15 066 (12.5) 16 410 (9.3) 10 548 (5.6) 4614 (4.7)

Abbreviation: BMI, body mass index (calculated as weight in kilograms divided by height in meters squared).

To conduct the analysis using FAR denominators, we used a Cox proportional hazards regression model to estimate hazard ratios (HRs) associating race/ethnicity with each outcome.35 We used conception, calculated by subtracting the gestational age at delivery from day of birth, as the origin of the time axis in the Cox proportional hazards regression model. By using conception as the origin instead of birth date, the denominator for event rate calculation at a given gestational age is ongoing pregnancies, the key feature of the FAR approach. Because of the limitations of hospital discharge data, we did not know the actual date of occurrence of each morbidity outcome and thus assumed the event date was the midpoint between birth and discharge. The event date, however, was known for deaths. Because infants who survive longer are at greater risk for morbidity, mortality must be taken into consideration in estimating differences in morbidity incidence. We considered death as a competing risk using the subdistribution hazard approach,36 which was applied in a study of infant morbidities.37 The goal of modeling mortality as a competing risk was to focus on the incidence of each morbidity. In contrast, a composite outcome of morbidity and mortality rates potentially obscures the differences in incidence among the morbidities studied. We conducted a sensitivity analysis that varied the assumption of the date of the morbid event to present a range of potential interval censoring bias; that is, models were repeated, assuming the date of the morbidity was at birth and also at discharge. We used a robust variance estimator to adjust the SE of model parameters to account for clustering of infants by hospitals.38 We censored infants born at 32 weeks’ gestation or later to estimate the associations between race/ethnicity and each outcome in VPTBs. We added sociodemographic and maternal covariates sequentially. Missing values for covariates were included as dummy variables in all multivariable models. Percentage missing ranged from 0.1% for parity to 0.7% for body mass index.

A third approach for addressing the issue of gestational-age stratification bias involves estimating the associations in the total population of births. Because the morbidities are specific to infants born very preterm, the clinical interpretation of this model for our research question is not clear. However, to allow for comparison with previous studies, we conducted a sensitivity analysis based on this approach. We also conducted a sensitivity analysis that changed the definition of NEC because of the well-known difficulties in diagnosing and coding this morbidity and to be consistent with previous literature. All analyses were conducted in SAS, version 9.4 (SAS Institute Inc). All study characteristic χ2 test of differences resulted in a 2-sided P value<. 001 for all comparisons.

Results

In total, 582 297 infants were included in our study. Of these infants, 285 006 were female (48.9%) and 297 291 were male (51.0%). Sample characteristics by race/ethnicity are shown in Table 1. Very preterm birth was most frequent among black infants (2231 of 120 842 [1.8%]), followed by Hispanic (1645 of 176 835 [0.9%]), Asian (566 of 97 458 [0.6%]), and white (1076 of 187 162 [0.6%]) infants. White and Asian mothers were older than black and Hispanic mothers; the highest rates of teenaged (<20 years) birth were among Hispanic (132 406 [9.0%]) and black (8509 [7.0%]) women, and the highest rates of birth to mothers aged 35 to 39 years were among Asian (18 694 [19.2%]) and white (42 795 [22.9%]) women. The percentage of white mothers with private insurance was 2 to 3 times higher (120 961 [64.6%]) compared with black (30 562 [25.3%]), Hispanic (34 076 [19.3%]), and Asian (37 392 [38.4%]) mothers. White mothers most frequently initiated prenatal care in the first trimester (150 203 [80.3%]). Black and Hispanic mothers had the highest prevalence of obesity, smoking, and hypertensive disorders. Asian mothers, followed by black and Hispanic mothers, had the highest prevalence of gestational diabetes.

In the analysis of the VPTB subcohort, risk ratios (RRs) unadjusted for gestational age were elevated among black and Hispanic infants for NEC (RR, 1.68; 95% CI, 1.26-2.24 vs RR, 1.54; 95% CI, 1.13-2.08) and BPD (RR, 1.64; 95% CI, 1.38-1.95 vs RR, 1.32; 95% CI, 1.10-1.60) (Table 2). In fully adjusted models, black infants compared with white infants had an increased risk only of BPD (adjusted risk ratio [aRR], 1.34: 95% CI, 1.09-1.64) and borderline increased risk of NEC (aRR, 1.39; 95% CI, 1.00-1.93). Hispanic infants had borderline increased risk of NEC (aRR, 1.39; 95% CI, 0.98-1.96), and Asian infants had increased risk of ROP (aRR, 1.85; 95% CI, 1.15-2.97). Disparities between Asian and white infants were present only for ROP in both unadjusted and fully adjusted models (adjusted RR, 1.85; 95% CI, 1.15-2.97).

Table 2. Unadjusted and Adjusted Risk Ratios for Race/Ethnicity and Subcohort Using Log-Binomial Regression.

Outcome by Race Risk Ratio (95% CI)
Unadjusted Adjusteda Adjustedb Adjustedc
Necrotizing enterocolitis
Black 1.68 (1.26-2.24) 1.60 (1.18-2.17) 1.52 (1.10-2.11) 1.39 (1.00-1.93)
Hispanic 1.54 (1.13-2.08) 1.49 (1.07-2.06) 1.41 (1.00-1.99) 1.39 (0.98-1.96)
Asian 1.19 (0.79-1.78) 1.12 (0.47-1.70) 1.10 (0.71-1.71) 1.13 (0.73-1.76)
White 1 [Reference] 1 [Reference] 1 [Reference] 1 [Reference]
Intraventricular hemorrhage
Black 1.03 (0.72-1.48) 1.00 (0.67-1.48) 0.93 (0.63-1.40) 0.79 (0.53-1.16)
Hispanic 1.32 (0.92-1.89) 1.27 (0.86-1.88) 1.15 (0.76-1.73) 1.08 (0.72-1.62)
Asian 0.81 (0.47-1.38) 0.78 (0.45-1.34) 0.76 (0.43-1.33) 0.80 (0.46-1.38)
White 1 [Reference] 1 [Reference] 1 [Reference] 1 [Reference]
Bronchopulmonary dysplasia
Black 1.64 (1.38-1.95) 1.61 (1.34-1.93) 1.52 (1.24-1.87) 1.34 (1.09-1.64)
Hispanic 1.32 (1.10-1.60) 1.28 (1.05-1.56) 1.21 (0.97-1.51) 1.17 (0.94-1.47)
Asian 0.77 (0.58-1.02) 0.70 (0.52-0.94) 0.72 (0.53-0.99) 0.76 (0.55-1.03)
White 1 [Reference] 1 [Reference] 1 [Reference] 1 [Reference]
Retinopathy of prematurity
Black 1.43 (0.98-2.09) 1.16 (0.78-1.73) 1.05 (0.69-1.58) 0.91 (0.61-1.37)
Hispanic 1.51 (1.02-2.23) 1.18 (0.78-1.79) 1.08 (0.70-1.67) 1.05 (0.67-1.61)
Asian 2.09 (1.34-3.26) 1.76 (1.10-2.81) 1.87 (1.16-3.01) 1.85 (1.15-2.97)
White 1 [Reference] 1 [Reference] 1 [Reference] 1 [Reference]
a

Adjusted for sociodemographic factors (maternal age, parity, educational level, insurance status, and nativity; infant sex).

b

Additionally adjusted for maternal morbidities (pregestational hypertension, gestational hypertension, pregestational diabetes, gestational diabetes, placenta previa, obesity, and smoking status).

c

Additionally adjusted for gestational age and birth weight z score.

In the model using FAR denominators, black and Hispanic mothers compared with white mothers were at a higher risk of all neonatal morbidities in unadjusted and adjusted models (Table 3). Adjusted subdistribution HRs for black compared with white infants ranged from 2.73 for IVH (95% CI, 1.63-4.57) to 4.43 for BPD (95% CI, 2.88-6.81). Subdistribution HRs for Hispanic compared with white infants were also elevated but smaller in magnitude than those for black compared with white infants. Asian infants were at similar risk as white women for NEC, IVH, and BPD but were at an increased risk for ROP (adjusted HR = 2.43; 95% CI, 1.43-4.11). Sensitivity analyses that reassigned the timing of morbidity at birth or at discharge produced very similar subdistribution HRs (eTable 4 in the Supplement). In the sensitivity analysis of racial/ethnic disparity RRs using all births, estimates unadjusted for gestational age were similar to FAR estimates and estimates fully adjusted for gestational age and birthweight z score were similar to VPTB subcohort estimates (eTable 5 in the Supplement). Sensitivity analysis using varying definitions of NEC showed HRs of similar magnitude (eTable 6 in the Supplement).

Table 3. Unadjusted and Adjusted Subdistribution Hazard Ratios for Race/Ethnicity and Neonatal Morbidities (Infants <32 Weeks’ Gestation) Using Fetuses-at-Risk Approach .

Outcome by Race Hazard Ratio (95% CI)
Unadjusted Adjusteda Adjustedb
Necrotizing enterocolitis
Black 5.14 (3.66-7.25) 5.02 (3.52-7.16) 4.40 (2.98-6.51)
Hispanic 2.52 (1.79-3.54) 2.60 (1.81-3.74) 2.52 (1.71-3.73)
Asian 1.27 (0.81-2.00) 1.35 (0.85-2.17) 1.41 (0.94-2.14)
White 1 [Reference] 1 [Reference] 1 [Reference]
Intraventricular hemorrhage
Black 3.14 (1.88-5.22) 3.09 (1.79-5.34) 2.73 (1.63-4.57)
Hispanic 2.15 (1.28-3.59)) 2.18 (1.26-3.79) 2.12 (1.28-3.52)
Asian 0.85 (0.49-1.46) 0.91 (0.54-1.54) 0.97 (0.59-1.60)
White 1 [Reference] 1 [Reference] 1 [Reference]
Bronchopulmonary dysplasia
Black 5.00 (2.94-8.53) 5.04 (3.11-8.16) 4.43 (2.88-6.81)
Hispanic 2.15 (1.42-3.27) 2.22 (1.50-3.3) 2.18 (1.53-3.12)
Asian 0.82 (0.54-1.24) 0.84 (0.57-1.22) 0.91 (0.63-1.31)
White 1 [Reference] 1 [Reference] 1 [Reference]
Retinopathy of prematurity
Black 4.37 (2.88-6.64) 3.70 (2.36-5.77) 2.98 (2.01-4.40)
Hispanic 2.46 (1.56-3.88) 2.09 (1.31-3.36) 1.99 (1.32-3.01)
Asian 2.22 (1.31-3.77) 2.12 (1.23-3.67) 2.43 (1.43-4.11)
White 1 [Reference] 1 [Reference] 1 [Reference]
a

Adjusted for sociodemographic factors (maternal age, parity, educational level, insurance status, and nativity; infant sex).

b

Additionally adjusted for maternal morbidities (pregestational hypertension, gestational hypertension, pregestational diabetes, gestational diabetes, placenta previa, obesity, and smoking status).

Discussion

Black and Hispanic infants compared with white infants had a 2- to 4-fold increased risk of the 4 severe neonatal morbidities, and Asian infants were at an increased risk of ROP. In addition, these disparities were underestimated in the VPTB subcohort. The most marked difference between the 2 approaches was found for IVH: disparities between black and white infants as well as Hispanic and white infants were found in the FAR analysis but not in the VPTB subcohort analysis. These findings suggest that current reports of racial/ethnic disparities in very preterm or preterm morbidities underestimate the true magnitude of disparity.

This study is not the first in the clinical literature on VPTB morbidity to caution that associations between prenatal factors and neonatal morbidities stratified by gestational age may be biased.39 Shulman et al39 demonstrated gestational age stratification bias for associations of preeclampsia with ROP, and their results showed a similar bias for black race/ethnicity. However, Shulman and colleagues39 reported only the total association of preeclampsia with ROP across gestational age as an alternative, whereas we used FAR denominators to estimate the total association of race/ethnicity with morbidities in infants born before 32 weeks’ gestation. In addition, we calculated the total association of race/ethnicity across gestational age in a sensitivity analysis and found that estimates adjusting for gestational age were similar to those found in the VPTB subcohort. In contrast, the estimates before controlling for gestational age were of similar magnitude to those from the FAR analysis. However, a measure of association for VPTB morbidities across gestational ages is not easily interpreted, whereas in the FAR approach, the interpretation is specific to morbidity cases occurring before 32 weeks.

Our finding that different methods produced different magnitudes of association raises questions about which approach is most appropriate. Researchers have suggested that the most appropriate method for studying neonatal outcomes depends on the research question.40,41 Investigations into prenatal characteristics, such as race/ethnicity, or other conditions or factors present prior to birth may be particularly biased in using the conventional approach of stratifying by gestational age. Analyses of factors that clearly occur after birth can possibly use a conventional approach. Examples of such questions that have been posited in the literature include whether NICU level is associated with rates of BPD42 and whether oxygen saturation level is associated with risk of ROP.43 However, if unmeasured confounders resulting in gestational age stratification bias continue to be associated with risk of the postnatal outcome, bias may still be present. To date, much of the research on VPTB morbidity comes from prospective cohort studies or neonatal networks selected by gestational age or birth weight of less than 1500 g and thus is subject to potential collider stratification bias.44,45,46 This approach may be appropriate for study of practices or quality initiatives in the neonatal intensive care unit, but caution should be used when interpreting the findings of investigations into risk factors for neonatal morbidity occurring before birth, including race/ethnicity.

The implications of underestimating disparities in neonatal morbidities are substantial. Our findings of disparities suggest that health care professionals should be more vigilant on behalf of black and Hispanic populations. Estimates of minimal increased risk for black and Hispanic infants, or even decreased risk, may also distract health care professionals from the work of reducing morbidity among these vulnerable groups and from the fact that more black and Hispanic children with these morbidities would require early intervention services or follow-up care. Another consequence of underestimating disparities is it may stimulate erroneous physiologic explanations for a morbidity advantage; a similar misinterpretation was previously drawn from early studies that showed increased survival among very low-birth-weight black infants. In addition, insights about how the origin of severe neonatal morbidities could give rise to an association with race/ethnicity might be missed if true associations are dismissed as small or null in studies of prenatal characteristics.

Strengths and Limitations

A strength of this study is that the large number of infant data and the precise effect estimates we were able to obtain allowed us to identify important increased risk of ROP among Asian infants, a population often left out of smaller studies. We also had rich information on maternal sociodemographic and medical characteristics.

The analytic approach of FAR has several important limitations. The FAR model does not estimate the direct effect of race/ethnicity on VPTB morbidities (the effect is not mediated by gestational age) and therefore does not directly answer the research question investigators wish to answer by stratifying by gestational age. Instead, it estimates the total effect of race/ethnicity on VPTB morbidities, including via gestational age, among VPTB infants. It can also be difficult to interpret for postnatal events, because birth is necessary for the morbidity to be revealed, and there is an ongoing debate in the epidemiologic literature on its interpretation in these situations.47 However, if racial/ethnic disparities are examined from the vantage point of the pregnancy and the assumption that the morbidities originate during fetal development, then evaluating the disparities using the FAR approach is an improvement over solely estimating the disparities’ conditioning on gestational age, which are subject to collider stratification bias and thus underestimate their true magnitude. Our data source has several limitations. First, we did not have the date of diagnosis of morbidities to be able to refine the FAR model, although sensitivity analyses revealed that even broad assumptions about date of diagnosis did not substantially affect the hazard estimates. Nonetheless, interval censoring differential by race/ethnicity has the potential to influence the reported coefficients in an unknown direction. Second, we did not have information on stillbirths, which may have affected our measures of association. Although measures of neonatal morbidities derived from the International Classification of Diseases, Ninth Revision, codes in hospitalization data are used widely in the literature, they may lack sensitivity or specificity. Finally, differences in coding practices across hospitals, should they coincide with the distribution of black or Hispanic patients, could potentially bias our findings. Future research might expand on these findings by including information on onset of morbidity and by investigating the disparities in neonatal morbidities associated with varying contexts such as geographic region, urban or rural setting, and hospital characteristics (eg, neonatal intensive care unit level and volume).

Conclusions

We identified sizable disparities between black and white infants as well as Hispanic and white infants in 4 severe neonatal morbidities among very preterm infants. Previous literature underestimated these disparities, sometimes coming to conclusions of a lack of increased risk or even reduced risk of morbidity among non-white infants, particularly Hispanic infants. Disparity in ROP between Asian and white infants was also revealed. Clinicians and researchers should be aware that reports of racial/ethnic disparities of VPTB morbidities stratified by gestational age may underestimate the true magnitude of the disparities.

Supplement.

eTable 1. Severe Neonatal Morbidity ICD-9 Diagnosis and Procedure Codes

eTable 2. ICD-9 Codes and Birth Certificate Variables for Ascertainment of Maternal Conditions

eTable 3. ICD-9 Codes for Congenital Anomalies

eTable 4. Unadjusted and Adjusted Subdistribution Hazard Ratios for Race/Ethnicity and Major Neonatal Morbidities Among Infants <32 Weeks Using Fetuses-at-Risk Approach, Sensitivity Analysis Varying Assumptions for Date of Diagnosis of Morbidity (n = 582 297)

eTable 5. Unadjusted and Adjusted Risk Ratios for Race/Ethnicity and Infant Morbidities in Total Population, New York City, 2010-2014 (n = 582 297)

eTable 6. Unadjusted and Adjusted Subdistribution Hazard Ratios for Race/Ethnicity and Necrotizing Enterocolitis (NEC) Among Infants <32 Weeks Using Fetuses-at-Risk Approach With Varying Definitions of NEC (n = 582 297)

References

  • 1.Schempf AH, Branum AM, Lukacs SL, Schoendorf KC. The contribution of preterm birth to the black-white infant mortality gap, 1990 and 2000. Am J Public Health. 2007;97(7):1255-1260. doi: 10.2105/AJPH.2006.093708 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Msall ME, Sobotka SA, Dmowska A, Hogan D, Sullivan M. Life Course Health Development Outcomes After Prematurity: Developing a Community, Clinical, and Translational Research Agenda to Optimize Health, Behavior, and Functioning. Handbook of Life Course Health Development. Cham, Switzerland: Springer; 2018:321-348. doi: 10.1007/978-3-319-47143-3_14 [DOI] [PubMed] [Google Scholar]
  • 3.Lorch SA. A decade of improvement in neonatal intensive care: how do we continue the momentum? JAMA Pediatr. 2017;171(3):e164395. doi: 10.1001/jamapediatrics.2016.4395 [DOI] [PubMed] [Google Scholar]
  • 4.Stoll BJ, Hansen NI, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012. JAMA. 2015;314(10):1039-1051. doi: 10.1001/jama.2015.10244 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Anderson JG, Rogers EE, Baer RJ, et al. Racial and ethnic disparities in preterm infant mortality and severe morbidity: a population-based study. Neonatology. 2018;113(1):44-54. doi: 10.1159/000480536 [DOI] [PubMed] [Google Scholar]
  • 6.Townsel C, Keller R, Kuo C-L, Campbell WA, Hussain N. Racial/ethnic disparities in morbidity and mortality for preterm neonates admitted to a tertiary neonatal intensive care unit [published online October 25, 2017]. J Racial Ethn Health Disparities. 2017. doi: 10.1007/s40615-017-0433-2 [DOI] [PubMed] [Google Scholar]
  • 7.Wallace ME, Mendola P, Kim SS, et al. Racial/ethnic differences in preterm perinatal outcomes. Am J Obstet Gynecol. 2017;216(3):306.e1-306.e12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Gephart SM, Spitzer AR, Effken JA, Dodd E, Halpern M, McGrath JM. Discrimination of GutCheck(NEC): a clinical risk index for necrotizing enterocolitis. J Perinatol. 2014;34(6):468-475. doi: 10.1038/jp.2014.37 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Neu J, Walker WA. Necrotizing enterocolitis. N Engl J Med. 2011;364(3):255-264. doi: 10.1056/NEJMra1005408 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Llanos AR, Moss ME, Pinzòn MC, Dye T, Sinkin RA, Kendig JW. Epidemiology of neonatal necrotising enterocolitis: a population-based study. Paediatr Perinat Epidemiol. 2002;16(4):342-349. doi: 10.1046/j.1365-3016.2002.00445.x [DOI] [PubMed] [Google Scholar]
  • 11.Petrova A, Mehta R, Anwar M, Hiatt M, Hegyi T. Impact of race and ethnicity on the outcome of preterm infants below 32 weeks gestation. J Perinatol. 2003;23(5):404-408. doi: 10.1038/sj.jp.7210934 [DOI] [PubMed] [Google Scholar]
  • 12.Dietze TR, Rose FF, Moore TA. Maternal variables associated with physiologic stress and perinatal complications in preterm infants. J Neonatal Perinatal Med. 2016;9(3):271-277. doi: 10.3233/NPM-16915134 [DOI] [PubMed] [Google Scholar]
  • 13.Downard CD, Grant SN, Maki AC, et al. Maternal cigarette smoking and the development of necrotizing enterocolitis. Pediatrics. 2012;130(1):78-82. doi: 10.1542/peds.2011-3808 [DOI] [PubMed] [Google Scholar]
  • 14.Howell EA, Janevic T, Hebert PL, Egorova NN, Balbierz A, Zeitlin J. Differences in morbidity and mortality rates in black, white, and Hispanic very preterm infants among New York City hospitals. JAMA Pediatr. 2018;172(3):269-277. doi: 10.1001/jamapediatrics.2017.4402 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Profit J, Gould JB, Bennett M, et al. Racial/ethnic disparity in NICU quality of care delivery. Pediatrics. 2017;140(3):e20170918. doi: 10.1542/peds.2017-0918 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Kantorowska A, Wei JC, Cohen RS, Lawrence RA, Gould JB, Lee HC. Impact of donor milk availability on breast milk use and necrotizing enterocolitis rates. Pediatrics. 2016;137(3):e20153123. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Ananth CV, Schisterman EF. Confounding, causality, and confusion: the role of intermediate variables in interpreting observational studies in obstetrics. Am J Obstet Gynecol. 2017;217(2):167-175. doi: 10.1016/j.ajog.2017.04.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Kramer MS, Papageorghiou A, Culhane J, et al. Challenges in defining and classifying the preterm birth syndrome. Am J Obstet Gynecol. 2012;206(2):108-112. doi: 10.1016/j.ajog.2011.10.864 [DOI] [PubMed] [Google Scholar]
  • 19.Kramer MS, Zhang X, Platt RW. Analyzing risks of adverse pregnancy outcomes. Am J Epidemiol. 2014;179(3):361-367. doi: 10.1093/aje/kwt285 [DOI] [PubMed] [Google Scholar]
  • 20.Yerushalmy J. Mother’s cigarette smoking and survival of infant. Am J Obstet Gynecol. 1964;88(4):505-518. doi: 10.1016/0002-9378(64)90509-5 [DOI] [PubMed] [Google Scholar]
  • 21.North AF Jr, MacDonald HM. Why are neonatal mortality rates lower in small black infants than in white infants in similar birth weight? J Pediatr. 1977;90(5):809-810. doi: 10.1016/S0022-3476(77)81258-4 [DOI] [PubMed] [Google Scholar]
  • 22.Joseph KS, Demissie K, Platt RW, Ananth CV, McCarthy BJ, Kramer MS. A parsimonious explanation for intersecting perinatal mortality curves: understanding the effects of race and of maternal smoking. BMC Pregnancy Childbirth. 2004;4(1):7. doi: 10.1186/1471-2393-4-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Loftin R, Chen A, Evans A, DeFranco E. Racial differences in gestational age-specific neonatal morbidity: further evidence for different gestational lengths. Am J Obstet Gynecol. 2012;206(3):259.e1-259.e6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.VanderWeele TJ, Mumford SL, Schisterman EF. Conditioning on intermediates in perinatal epidemiology. Epidemiology. 2012;23(1):1-9. doi: 10.1097/EDE.0b013e31823aca5d [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Auger N, Gilbert NL, Naimi AI, Kaufman JS. Fetuses-at-risk, to avoid paradoxical associations at early gestational ages: extension to preterm infant mortality. Int J Epidemiol. 2014;43(4):1154-1162. doi: 10.1093/ije/dyu011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Auger N, Naimi AI, Fraser WD, et al. Three alternative methods to resolve paradoxical associations of exposures before term. Eur J Epidemiol. 2016;31(10):1011-1019. doi: 10.1007/s10654-016-0175-1 [DOI] [PubMed] [Google Scholar]
  • 27.Joseph KS. A consilience of inductions supports the extended fetuses-at-risk model. Paediatr Perinat Epidemiol. 2016;30(1):11-17. doi: 10.1111/ppe.12260 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Yudkin PL, Wood L, Redman CW. Risk of unexplained stillbirth at different gestational ages. Lancet. 1987;1(8543):1192-1194. [DOI] [PubMed] [Google Scholar]
  • 29.Korst LM, Fridman M, Lu MC, et al. Monitoring childbirth morbidity using hospital discharge data: further development and application of a composite measure. Am J Obstet Gynecol. 2014;211(3):268.e1-268.e16. doi: 10.1016/j.ajog.2014.03.011 [DOI] [PubMed] [Google Scholar]
  • 30.Ford JB, Roberts CL, Algert CS, Bowen JR, Bajuk B, Henderson-Smart DJ; NICUS group . Using hospital discharge data for determining neonatal morbidity and mortality: a validation study. BMC Health Serv Res. 2007;7(1):188. doi: 10.1186/1472-6963-7-188 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Jensen EA, Lorch SA. Effects of a birth hospital’s neonatal intensive care unit level and annual volume of very low-birth-weight infant deliveries on morbidity and mortality. JAMA Pediatr. 2015;169(8):e151906. doi: 10.1001/jamapediatrics.2015.1906 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Lain SJ, Hadfield RM, Raynes-Greenow CH, et al. Quality of data in perinatal population health databases: a systematic review. Med Care. 2012;50(4):e7-e20. doi: 10.1097/MLR.0b013e31821d2b1d [DOI] [PubMed] [Google Scholar]
  • 33.Spiegelman D, Hertzmark E. Easy SAS calculations for risk or prevalence ratios and differences. Am J Epidemiol. 2005;162(3):199-200. doi: 10.1093/aje/kwi188 [DOI] [PubMed] [Google Scholar]
  • 34.Schisterman EF, Whitcomb BW, Mumford SL, Platt RW. Z-scores and the birthweight paradox. Paediatr Perinat Epidemiol. 2009;23(5):403-413. doi: 10.1111/j.1365-3016.2009.01054.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Platt RW, Joseph KS, Ananth CV, Grondines J, Abrahamowicz M, Kramer MS. A proportional hazards model with time-dependent covariates and time-varying effects for analysis of fetal and infant death. Am J Epidemiol. 2004;160(3):199-206. doi: 10.1093/aje/kwh201 [DOI] [PubMed] [Google Scholar]
  • 36.Fine JP, Gray RJ. A proportional hazards model for the subdistribution of a competing risk. J Am Stat Assoc. 1999;94(446):496-509. doi: 10.1080/01621459.1999.10474144 [DOI] [Google Scholar]
  • 37.Patel RM, Knezevic A, Shenvi N, et al. Association of red blood cell transfusion, anemia, and necrotizing enterocolitis in very low-birth-weight infants. JAMA. 2016;315(9):889-897. doi: 10.1001/jama.2016.1204 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Wei L-J, Lin DY, Weissfeld L. Regression analysis of multivariate incomplete failure time data by modeling marginal distributions. J Am Stat Assoc. 1989;84(408):1065-1073. doi: 10.1080/01621459.1989.10478873 [DOI] [Google Scholar]
  • 39.Shulman JP, Weng C, Wilkes J, Greene T, Hartnett ME. Association of maternal preeclampsia with infant risk of premature birth and retinopathy of prematurity. JAMA Ophthalmol. 2017;135(9):947-953. doi: 10.1001/jamaophthalmol.2017.2697 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Hutcheon JA, Platt RW. The missing data problem in birth weight percentiles and thresholds for “small-for-gestational-age.” Am J Epidemiol. 2008;167(7):786-792. doi: 10.1093/aje/kwm327 [DOI] [PubMed] [Google Scholar]
  • 41.Schisterman EF, Sjaarda LA. No right answers without knowing your question. Paediatr Perinat Epidemiol. 2016;30(1):20-22. doi: 10.1111/ppe.12258 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Lapcharoensap W, Gage SC, Kan P, et al. Hospital variation and risk factors for bronchopulmonary dysplasia in a population-based cohort. JAMA Pediatr. 2015;169(2):e143676. doi: 10.1001/jamapediatrics.2014.3676 [DOI] [PubMed] [Google Scholar]
  • 43.Manja V, Lakshminrusimha S, Cook DJ. Oxygen saturation target range for extremely preterm infants: a systematic review and meta-analysis. JAMA Pediatr. 2015;169(4):332-340. doi: 10.1001/jamapediatrics.2014.3307 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Horbar JD, Edwards EM, Greenberg LT, et al. Variation in performance of neonatal intensive care units in the United States. JAMA Pediatr. 2017;171(3):e164396. doi: 10.1001/jamapediatrics.2016.4396 [DOI] [PubMed] [Google Scholar]
  • 45.Rysavy MA, Li L, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Between-hospital variation in treatment and outcomes in extremely preterm infants. N Engl J Med. 2015;372(19):1801-1811. doi: 10.1056/NEJMoa1410689 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Zeitlin J, Manktelow BN, Piedvache A, et al. ; EPICE Research Group . Use of evidence based practices to improve survival without severe morbidity for very preterm infants: results from the EPICE population based cohort. BMJ. 2016;354:i2976. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Harmon QE, Basso O, Weinberg CR, Wilcox AJ. Two denominators for one numerator: the example of neonatal mortality. [published online March 7, 2018]. Eur J Epidemiol. 2018;33(6):523-530. doi: 10.1007/s10654-018-0373-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement.

eTable 1. Severe Neonatal Morbidity ICD-9 Diagnosis and Procedure Codes

eTable 2. ICD-9 Codes and Birth Certificate Variables for Ascertainment of Maternal Conditions

eTable 3. ICD-9 Codes for Congenital Anomalies

eTable 4. Unadjusted and Adjusted Subdistribution Hazard Ratios for Race/Ethnicity and Major Neonatal Morbidities Among Infants <32 Weeks Using Fetuses-at-Risk Approach, Sensitivity Analysis Varying Assumptions for Date of Diagnosis of Morbidity (n = 582 297)

eTable 5. Unadjusted and Adjusted Risk Ratios for Race/Ethnicity and Infant Morbidities in Total Population, New York City, 2010-2014 (n = 582 297)

eTable 6. Unadjusted and Adjusted Subdistribution Hazard Ratios for Race/Ethnicity and Necrotizing Enterocolitis (NEC) Among Infants <32 Weeks Using Fetuses-at-Risk Approach With Varying Definitions of NEC (n = 582 297)


Articles from JAMA Pediatrics are provided here courtesy of American Medical Association

RESOURCES