Methodological Issues in the Design and Analyses of Neonatal Research Studies: Experience of the NICHD Neonatal Research Network

Abstract

Objective

Impressive advances in neonatology have occurred over the 30 years of life of The Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network (NRN). However, substantial room for improvement remains in investigating and further developing the evidence base for improving outcomes among the extremely premature. We discuss some of the specific methodological challenges in the statistical design and analysis of randomized trials and observational studies in this population.

Findings

Challenges faced by the NRN include designing trials for unusual or rare outcomes, accounting for and explaining center variations, identifying other subgroup differences, and balancing safety and efficacy concerns between short-term hospital outcomes and longer term neurodevelopmental outcomes.

Conclusions

The constellation of unique patient characteristics in neonates calls for broad understanding and careful consideration of the issues identified in this paper for conducting rigorous studies in this population.

INTRODUCTION

Impressive advances in neonatology have occurred over the 30 years of life of The Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network (NRN). However, improvement in survival for extremely premature babies has plateaued in recent years despite more aggressive use of antenatal steroids, antibiotics, and surfactant, while in-hospital morbidities, such as bronchopulmonary dysplasia (BPD), retinopathy of prematurity (ROP), intracranial hemorrhage (ICH), and sepsis remain high^1–3 as premature births have increased.⁴ Although this has led to an increase in the number of infants at higher risk for long-term neurodevelopmental impairment (NDI), most investigators have reported that rates of neurologically intact survival (among live births) at 18 months to 2 years remain unchanged. Allowing for the complexities in interpreting a composite outcome (survival free of NDI) discussed later, overall this illustrates both the progress made in improving outcomes and the substantial room for improvement that remains.^5,6 These trends collectively have widespread implications for health care delivery and point to the continued need for targeted and rigorous research to develop better treatment and management strategies for neonates that improve long-term rehabilitation and outcome.

Reducing the high rates of in-hospital morbidity and later NDI among extremely premature infants remains a significant public health challenge, highlighting the importance of ongoing evidence-based research in this area. In addition, although late preterm births currently account for 75% of all neonatal intensive care unit (NICU) admissions, little evidence-based research occurs for these infants.^7,8 Thus, critical gaps in neonatal research remain, and many of the more severe diseases (such as necrotizing enterocolitis [NEC] or neonatal encephalopathy) are relatively infrequent conditions that require multicenter involvement to study them. Significant methodological challenges also exist in designing rigorous trials for unusual or rare outcomes, accounting for and explaining center variations, identifying other subgroup differences, and balancing safety and efficacy concerns between short-term hospital outcomes and longer term neurodevelopmental outcomes. These challenges require innovative trial design and analysis strategies to address them. Over the past 30 years of its existence, the NRN has conducted important studies to fill critical evidence gaps in the field and tackled several methodological issues in study design and data analyses in this area (see Table 1). The following discussion briefly highlights some of the special methodological concerns in neonatal studies and the NRN experience in addressing them.

Table 1.

Selection of Major NRN Studies Conducted in the Last 10 Years

Study Title*	Statistical Design Features	Statistical Analysis Features
Active Randomized Clinical Trials
Hydrocortisone for BPD	Powered for joint evaluation of efficacy conditional on safety	Two-stage sequential evaluation of efficacy conditional on safety
Late Hypothermia	RCT stratified by age of enrollment and level of encephalopathy	Bayesian interim monitoring for safety, futility, and efficacy Bayesian analysis of final outcomes for treatment benefit
MILK Trial	RCT stratified by birth weight (BW), center and feeding group eligibility Simulations for operating characteristics	Linear regression to estimate the adjusted mean difference in cognitive scaled scores between the two treatment groups
NEC Surgery Trial (NEST)	Comprehensive cohort design Randomization stratified by predicted baseline risk	Meta-analysis of randomized and preference cohorts after propensity modeling/adjustment in the latter
Optimizing Cooling	2×2 factorial design with stratified randomization	Marginal and within-table adjusted analyses
Transfusion of Prematures (TOP)	Pragmatic RCT with stratified randomization and pilot study of blood bank variations	Survival analyses of death Longitudinal modeling of growth Economic evaluation
Hydrocortisone for Cardiovascular Insufficiency	Randomized, multicenter, double- blind, placebo-controlled trial Patients are enrolled and randomized in a variable block design, 1:1 ratio of hydrocortisone or placebo	Planned sample size reanalysis after 1 year of enrollment based on overall outcome rate
Preemie Hypothermia	RCT stratified by age of enrollment and level of encephalopathy	Bayesian interim monitoring for safety, futility, and efficacy Bayesian analysis of final outcomes for treatment benefit
Incubator Weaning	RCT stratified by center and gestational age	Median regression adjusting for the trial stratification factors
Inositol RCT	RCT stratified by center and gestational age	Separate analyses plans for publication and FDA submission
Completed Randomized Clinical Trials
Early BP Pilot	Time-limited feasibility pilot study with 2×2 factorial design	Feasibility analyses
Vitamin E Pilot	Pilot, double-blind placebo- controlled RCT	Analyses of Vitamin E levels over time
SUPPORT	2×2 factorial design, randomization stratified by site and GA groups	Marginal outcomes analyses Interim group sequential monitoring in factorial setup Familial clustering
Inositol Multidose Trial	Double-blind phase II trial conducted under IND from FDA in three treatment groups stratified by GA	PK modeling allowing for endogenous Inositol synthesis and change from IV to oral administration during the intervention Used to establish dosage for pivotal phase III trial
Phototherapy	RCT stratified by site and BW groups	Primary outcome at 18 months (used surrogate markers for interim safety/efficacy monitoring)
Registry Studies
Generic Data Base (GDB)	Registry of all NRN very low BW births	Varied analytic approaches, including multiple regression, Classification and Regression Trees, longitudinal analyses, multilevel hierarchical generalized linear models, prediction modeling, and genetic epidemiology analyses
22–26 Month Follow-Up	Registry of all NRN extremely low BW outcomes at 18 months
Moderate Preterm Registry	Registry of moderate preterm births
Term Control Registry
Active Observational Studies
ALPS aEEG	Prospective, cohort study
Early Onset Sepsis II	Surveillance study	Incidence/prevalence rates
Completed Observational Studies
Term Hypotension	Observational, time-limited study	Linear and logistic regression
Early Blood Pressure	Observational, time-limited study	BP profiles analyses
Early Onset Sepsis	Surveillance study	Incidence/prevalence rates
Preemie aEEG	Prospective cohort study Sample size based on false- negative rate	Interobserver reliability modeling Prediction modeling
Candida	Prospective cohort study	Likelihood ratio–based risk stratification, latent class analyses for diagnostic tests
PCV-7	Vaccine efficacy study	Multiple linear/logistic regression for geometric mean titers

STATISTICAL CONSIDERATIONS FOR NEONATAL RESEARCH

Designing studies in neonatal populations involves particular challenges, some of which apply almost universally across all NRN studies (e.g., the need to balance both proximal and distal outcomes to evaluate safety and efficacy), and some of which arise during the planning of specific studies (e.g., switching drug administration mode from IV to oral as an infant matures in a pharmacokinetics [PK] study). In this section, we give examples of statistical innovations we have used to overcome specific challenges in recent NRN studies, and we describe methods emerging from the research of statisticians at RTI and elsewhere that can be applied to challenges we foresee for future studies. The approaches discussed here augment standard statistical techniques used in more straightforward studies and analyses, which are not discussed here.

Competing Outcomes

An important issue in designing randomized clinical trials (RCTs) for high-risk patients is the selection of an appropriate primary outcome when death is a competing outcome. In this situation, some patients will die before the outcome that the intervention is expected to prevent can be diagnosed, e.g. ROP. For this reason, the primary outcome in such trials is often a composite outcome, e.g., Death or BPD even if the intervention is not expected to affect mortality.

As illustrated by the Network SUPPORT trial, an important advantage of including deaths in the primary outcome is that mortality might unexpectedly be affected. This trial assessed whether use of an oxygen saturation goal in the lower half of the recommended range would reduce severe ROP among infants born at 24 to 27 weeks gestation. Based on the best available evidence before the trial, no effect on mortality was hypothesized or expected by the investigators, contrary to what critics unfamiliar with the issue of competing outcomes have assumed about this study.⁹ The lower saturation goal did reduce severe ROP. However, this benefit was offset by an unexpected increase in death with no significant effect on the primary outcome of death or severe ROP. Had the primary outcome been severe ROP alone (among survivors), the primary outcome would not have captured the most important effect of the lower saturation goal, and the finding a mortality difference that prompted the recommendation to use a high saturation goal.¹⁰

In general, unless death is part of the primary outcome for a trial in a population that is likely to experience a substantially high death rate before the true outcome of interest can be assessed (again, because death is a competing outcome for any later morbidity in the NICU population), interpreting downstream events is complicated and inferences about the effects of treatment on these risks may be biased. This is because differential death rates in the comparison groups makes the survivors in these two groups a non-random sample of the randomized population, resulting in a biased and non-randomized comparison of outcome rates.

In addition to accounting for mortality, certain composite outcomes in neonatal research have considerable public health significance in their own right. Since the survival of extremely premature babies with profound impairment often has lifelong significance for these children, their families and society at large, survival free of neurodevelopmental impairment (or, neurologically intact survival) is a clinically meaningful outcome in its own right that is frequently used as a primary outcome for many NRN trials. For example, the ongoing Transfusion of Prematures (TOP) trial aims to examine whether the clinically relevant composite primary outcome of death or significant neurodevelopmental impairment in survivors at 22–26 months of corrected age is less common among preterm infants who, by transfusion practice, are maintained at higher hemoglobin levels.¹¹

Although competing outcomes are an unavoidable problem, some investigators resist the use of composite outcomes, including those that include death as a competing outcome, because outcomes of differing importance are given equal weight. In principle, this is not an inherent problem for composite outcomes. A potential solution is to weight the different outcomes according to their importance (utility) as judged by patients (or their family members) who know the most about or are most likely to experience these outcomes.^12–14 However difficult this approach may seem, in the long run it is likely to be necessary to resolve not only how different components of a composite outcome should be weighted but also the broader problem of determining when the benefits of treatment outweigh the hazards.

Accounting for competing and composite outcomes in primary outcome selection, sample size/power calculations, and interim monitoring plans is often necessary for neonatal trials. The NRN has helped develop innovative approaches to this issue, illustrated by the design of a current RCT to evaluate the use of hydrocortisone to reduce BPD.¹⁵ The trial seeks to test whether a 10-day tapering course of hydrocortisone treatment for infants <30 weeks estimated gestational age at birth who remain intubated at 14–28 days postnatal age can reduce the composite outcome of BPD or death at 36 weeks PMA. However, because of safety concerns for hydrocortisone, particularly with respect to neurodevelopment, the intervention will not be considered successful if it increased the rates of NDI or death at follow-up. Thus, the primary outcome for this study includes both a measure of efficacy (improvement in survival without physiologically defined moderate to severe BPD) and safety (survival without moderate or severe NDI at 22–26 months corrected age). Consequently, we designed the study to provide a sequential evaluation of the composite hypothesis that (i) administration of hydrocortisone has efficacy benefit in that it reduces the risk of death or BPD at 36 weeks, and (ii) administration of hydrocortisone has an acceptable long-term safety profile with respect to death or NDI at 22–26 months.

This composite hypothesis will be evaluated sequentially. First, we will test the hypothesis that infants treated with hydrocortisone have a lower risk of death or BPD at 36 weeks than infants treated with placebo. If this test of efficacy indicates benefit of the hydrocortisone arm, then the safety of the treatment will be evaluated descriptively through an assessment of the risk to benefit ratio of the hydrocortisone treatment. Specifically, this safety outcome will be considered a “success” if either (1) the point estimate of risk of death/NDI is lower on the hydrocortisone arm than on the control arm, or (2) there is an increase in risk on the hydrocortisone arm, but the lower limit of a one-sided 95% confidence interval for the ratio of increased benefit for BPD to increased risk for NDI is greater than 4. In other words, for every additional four infants surviving without BPD, no more than one would experience death/NDI. Sample size calculations for this trial accounted for this design feature by first using standard power analysis to assess the sample size required to demonstrate a difference in death or BPD at 36 weeks on the two treatment arms. Second, simulation analyses were used to assess the probability of success for the safety outcome based on either of the two criteria defined above. The empirical probability of success for the descriptive analyses used to evaluate the safety outcome is analogous to the power of a formal hypothesis test.

Although the most common primary outcome in NRN studies is death or NDI, we anticipate future studies in which NDI or cognitive performance among survivors may be of primary interest. One possible analytical approach is to treat deaths as informatively missing and compare treatments within principal strata defined by potential survival outcomes.¹⁶ As appropriate, other emerging methods for the analysis of semicompeting risks such as semiparametric regression models,¹⁷ illness–death models with shared frailty,¹⁸ and multistate modeling in which individuals pass through various states (e.g., impairment and death, with interval censoring to accommodate preplanned evaluation times)¹⁹ may be considered.

Trials for Conditions with Low Prevalence

Neonatal conditions such as NEC, severe respiratory failure requiring extracorporeal membrane oxygenation, and perinatal hypoxiaischemia have low prevalence but severe consequences. Although RCTs are the gold standard in clinical research, attaining sufficient enrollment for an adequately powered RCT in a reasonable amount of time can be challenging, and randomizing patients is not always feasible. A number of approaches have been used by NRN trials to address this recurring issue.

1. Comprehensive Cohort Design. One approach used in the ongoing NEC Surgery Trial comparing death or NDI for treatment with initial drain versus initial laparotomy in extremely premature babies with NEC was to implement a comprehensive cohort design²⁰ to include both randomized and nonrandomized (physician preference) patients, with causal inference techniques used for the latter.^21,22 Comprehensive cohort designs may be useful in judging whether treatment effects on the risk-adjusted outcomes of eligible nonrandomized patients are comparable to those among randomized patients. However, this assessment requires that all or virtually all eligible nonrandomized infants, including those who die, be included to avoid selection bias. Ideally, their outcomes would be collected as part of routine clinical surveillance without requiring specific parental consent for inclusion in the analyses.

2. Bayesian Inference. Rare conditions and subgroup analyses will continue to pose a challenge for future NRN studies. In such settings with small sample sizes, Bayesian methods can be used to estimate the probability of treatment benefit and harm even for a conventionally underpowered trial.²³ These probabilities cannot be calculated with traditional frequentist methods, and Bayesian analyses are needed to calculate them and the probability of a clinically important treatment effect.²⁴

   For interventions where prior studies in the area exist, as is the case of the ongoing Late Hypothermia and Preemie Hypothermia trials, use of this prior information can potentially reduce the required sample size compared with a frequentist design. Additionally, by incorporating the current state of knowledge through the use of informative prior distributions, Bayesian analyses provide a natural way to formally incorporate empirical evidence on the plausible magnitude of treatment effects and have been proposed as a way to temper large effects that may be spurious.^25,26 Prior information that was skeptical of large treatment effects was used for the design of the NRN Late Hypothermia and Preemie Hypothermia trials and for the analyses of the NRN Aggressive Phototherapy Trial.^27,28

   If prior evidence or even strong beliefs from investigators exist for a particular intervention, then enthusiastic or skeptical priors can be used as a sensitivity analysis to ensure that trial results are robust to differing degrees of prior belief regarding the efficacy of the intervention. The resulting final or posterior distributions for the treatment effect may differ for different levels of prior belief, particularly for small sample sizes, and investigators may reach different conclusions for a particular study. Although this subjectivity of prior belief is often offered as a main drawback of a Bayesian approach, it can also be seen as an advantage because it formalizes how experts with differing preexisting opinions will view the results. Different prior distributions and all other aspects of a Bayesian trial design and analysis plan should be evaluated with simulation studies (using different scenarios to represent a range of potential treatment effects) to ensure optimal trial operating characteristics (i.e., adequate control for type I and II errors).^29,30

3. Other Designs. Future studies may also take advantage of emerging study designs for minimally sized studies. Group sequential designs reduce sample size by stopping studies early if effect sizes differ from what is expected, and internal pilot/adaptive designs³¹ use interim power analysis to adjust the sample size for misspecification of parameters, while internal pilot with interim analysis designs combine the advantages of both.³²

Randomization

Proper randomization, conducted carefully and with adequate thought, is essential for preserving the integrity of any RCT. Trials in neonates, however, often present special issues for consideration, some of which are discussed below.

• Multiple Births. Multiple births are fairly common in the extremely premature population that is the subject of most NRN trials, with more than a quarter of babies born at 22–28 weeks gestation reported to fit this description.¹ Because outcomes of infants from the same birth may be correlated because of shared environmental and genetic factors, the statistical question then concerns whether babies from the same birth should be randomized individually or together (i.e., whether to randomize each baby or each mother/pregnancy). The advantage of the latter approach, which is akin to a cluster randomized trial (with the cluster here being the mother) is that it forces the analysis to account for any correlation in outcomes among babies from the same birth. This is also the natural approach to adopt if the randomized intervention being tested will be conducted on the mother antenatally or in the delivery room, thus ensuring that babies from the same birth are, by design, exposed to the same intervention.

• For interventions conducted on babies after birth, however, cluster randomization of multiples to the same treatment arm can present substantial downsides. First, the correlation in outcomes between multiples may not be especially high because less than 10% of multiple births in the very low birth weight population are expected to be monozygotic, and thus share identical genetic makeup.^33,34 So, although monozygotic multiples may be expected to have higher correlation in outcomes, for dyzygotic multiples other factors such as gender, birth weight and sex discordance, and events during the NICU course may be far more important in determining outcomes.^35–37 Thus, cluster randomization may be trying to solve a problem that may be substantively insignificant.

• Second, any small correlation in outcomes, if present among multiple births, can be accounted for in the analysis using traditional methods for correlated data analysis (such as generalized estimating equation [GEE] or random effects models) without recourse to group randomization of multiples. It is interesting to note that one of the few publications in this area did not note any advantages to specialized modeling for correlated data in the case of binary outcomes (the type of outcomes most commonly used for large multicenter neonatal trials),³³ although others have reported different results.^38–40

• Third, cluster randomization entails the use of mixed models or GEE for their analysis, regardless of whether any appreciable correlation is actually present in the outcomes of interest. Such methods, by having to estimate additional covariance parameters from the same set of data, usually produce estimates that are less precise with wider confidence intervals, compared to ordinary regression models that do not account for correlations. It is also noteworthy that in cluster randomization the unit of randomization (the cluster or the mother) is different from the unit of analysis and inference (the baby), and thus inferences have to be tailored and interpreted accordingly.

• Fourth, it is often difficult to develop precise sample size and power calculations for a cluster randomized trial because of the lack of prior data on the size of the intracluster correlation for different outcomes (which would vary depending on the outcome, the composition of multiple births in the total sample, and the composition of monozygotic births among the multiple births). Typically, accounting for this correlation increases the sample size required, which has implications for both trial budgets and the amount of time needed to finish accrual.

• In summary, although group randomization of multiple births may be conceptually appealing, there are a number of methodologic issues that should be carefully considered before a decision is made to use this approach. The NRN has used individual randomization for almost all its randomized trials, with group randomization only used for the SUPPORT trial, which sought antenatal consent, and where one of the two interventions took place in the delivery room.⁴¹ We recognize that ethical issues such as parent preference should also be strongly considered in the trial design process, although these are outside the scope of this report.⁴² This remains a fertile area for further methodological research that can potentially use comprehensive simulations over a wide range of scenarios to conduct a statistical cost-benefit analysis of the relative merits of either approach to randomization in the population of extremely premature babies.

• Clinic Interventions. Sometimes, interventions need to be tested at the clinic (or NICU) level because they involve broad-based unit practice or process changes, or interventions that cannot be individually delivered to patients. Even if the interventions can be theoretically tailored individually, sometimes that is not possible in reality without “contaminating” the practice across other unit clinical staff, in which case again the intervention needs to be tested at the unit level. Such interventions are tested in randomized trials using cluster randomization where the clinics or NICUs, rather than individual babies, are randomized to different treatment arms.⁴³ Because the unit of randomization for these trials is the center (or NICU), the power and sample size requirements and the available precision for treatment effect estimates for these trials is driven more by the total number of centers in the trial than the total number of babies enrolled.

• A variation on cluster randomization involves cluster randomized crossover trials where each cluster (or clinic) receives each intervention at least once in separate periods of time.^44,45 During each time period the cluster may contain different babies, the same babies, or a mixture of both different and same babies.⁴⁶ Such designs, if practically feasible, are typically more efficient and require less sample size than traditional cluster randomized designs if the cluster environment remains similar between time periods. The stepped wedge cluster randomized trial is another novel variation on the crossover cluster design, which involves random and sequential crossover of clusters from control to intervention until all clusters are exposed. It is increasingly being used to study Phase IV–type effectiveness interventions, where efficacy has already been established and it would be ethically problematic not to offer the intervention to all clusters, but logistically challenging to start the intervention at many sites simultaneously.⁴⁷

• With the exception of the cluster randomized Benchmarking trial,⁴⁸ such trials have been rare in the NRN because of the limited number of centers available to randomize, which makes it difficult to estimate treatment effects with adequate precision. The statistical challenges outlined for cluster randomized trials in the previous discussion all apply here as well. However, cluster randomized trials remain the most rigorous means by which to study interventions instituted at the center level.

Multiple Interventions: Factorial Designs

At times, testing the efficacy of two or more treatments concurrently may help determine whether there is any advantage in using them together and use resources more efficiently by studying two treatments in the same trial.⁴⁹ However, efficiency can only be achieved for factorial designs if one can reasonably assume that there is no statistical interaction between the different treatments being tested (i.e., the effect of one is not changed in the presence of the other). Factorial designs also provide the most rigorous means to test for the presence of statistical interactions among the treatments being studied. However, the goals of efficiency and studying interactions are in conflict, and one can only study interactions in a two-treatment (or 2×2) factorial design with adequate power if the sample size is roughly four times higher than what would be required under an assumption of no statistical interaction. To be studied together in a factorial design, certain conditions need to hold:

1. It is possible to administer the treatments/interventions together without having to modify one because of the presence of the other.

2. All possible treatment group combinations, including placebo, are ethically acceptable.

3. All treatment group combinations are of scientific interest.

4. The different treatments being tested do not have the same mechanism of action (because otherwise either one will answer the scientific question).

When the primary interest in conducting a factorial study is to test the individual effects of two treatments or interventions together simply for efficiency, and not to study their statistical interaction, an understanding of how the two treatments jointly affect the outcome of interest in comparison to how they independently affect outcomes is still essential for appropriate power and sample size calculations. Clinically, the no interaction assumption often used in a factorial design means that the treatment effect of Treatment A in a population of individuals already treated with Treatment B is identical to the effect of Treatment A when no other treatment has been implemented. Clinicians often feel that such an assumption is not tenable in that a second treatment often achieves diminished results if some benefit is achieved by application of another treatment. Failure to account for this potential diminished return (typically characterized as a sub-additive interaction effect) will result in an underpowered factorial design.

In the absence of historic information about statistical interactions between different treatments being studied, and in recognition of the prohibitive sample size required to examine such interactions with adequate statistical power, in general factorial design trials in the NRN have assumed no independence among the studied treatments, but tested for the presence of such interactions once the trials were complete. Earlier NRN trials have tested permissive ventilation and steroids in reducing BPD,^50,51 and delivery room CPAP or surfactant to reduce BPD, in conjunction with different oxygen saturation targets to reduce ROP.^10,41 More recently, the Optimizing Cooling trial tested the efficacy of longer and deeper cooling in improving neurodevelopmental outcomes of neonatal encephalopathy using a 2×2 factorial design.⁵²

Statistical Interim Monitoring and Stopping Rules

Most NRN trials have a composite primary endpoint of death or morbidity, which complicates safety monitoring when trials can potentially be stopped early because of differences in mortality at interim analyses while the composite outcome will be tested at the end of the trial.⁵³ These trials require interim monitoring strategies with appropriate partitioning of the overall Type I error rate. For morbidities such as NEC that may manifest over time, survival analysis can be used to model time to onset of disease, with deaths censored.⁵⁴ Typically, to ensure patient safety, we monitor adverse events more frequently (e.g., every 100 babies for the ongoing Hydrocortisone trial) than measures of efficacy (e.g., 50% and 75% of primary outcome accrual for the same trial), and use relatively liberal Pocock statistical bounds to ensure interim safety and more conservative O’Brien Fleming bounds to determine interim efficacy.

Although the Hydrocortisone trial provides an example of a typical approach to interim testing, monitoring plans are tailored to each individual study. For example, in the ongoing Preemie Hypothermia trial designed using Bayesian principles, interim safety is monitored using posterior probability of treatment harm (based on a neutral prior probability of treatment benefit), while interim efficacy and futility are conservatively monitored using the posterior probability of treatment benefit (based on a skeptical prior probability of treatment benefit for efficacy monitoring, and an enthusiastic prior probability of treatment benefit for futility monitoring).

Bayesian monitoring approaches have potential advantages^29,55,56 that include incorporation of results from prior trials to better assess the likelihood of treatment benefit or harm and ensure that treatment recommendations are well justified based on all relevant trials.⁵⁷ For monitoring of trials, at a given interim analysis the posterior probability of the treatment effect is computed from the prior probability and the interim data. Stopping guidelines need to be prespecified for each type of interim monitoring. When monitoring a trial for efficacy, a large posterior probability threshold (i.e., > 97%) would be required to stop a trial early for benefit. Conversely, when considering stopping for futility, a stopping guideline would require that the probability of treatment benefit is very low (i.e., < 10%). The exact probability thresholds will differ for different interventions and patient populations. For example, a particularly high probability of a clinically meaningful benefit might be required for therapies that are invasive, hazardous, or extremely expensive. Thus, the appropriate stopping probability thresholds need to be discussed between the statistician and clinical investigators for each study.

A Bayesian approach can also incorporate a wide range of viewpoints and indicate the magnitude of the difference between treatment groups that would be needed at the end of the trial to convince those who are skeptical and those who were enthusiastic about the value of the therapy prior to the trial.^24,57 For example, to stop early for benefit a Bayesian analysis would use a prior skeptical of treatment benefit. The resulting posterior probability should be sufficient to convince an investigator who was skeptical of any treatment benefit at the beginning of the trial that the therapy is beneficial. To stop for futility, we would take the position of an enthusiastic investigator with strong prior belief in treatment benefit and assess whether there is sufficiently convincing evidence that there is little chance of benefit from the intervention.

When deciding whether to continue recruitment, pause, or terminate a trial, a Data and Safety Monitoring Committee (DSMC) can then not only weigh the current evidence for benefit, harm, or futility from the posterior probability but also formally assess how these results would be viewed by clinicians with different levels of skepticism or enthusiasm. A DSMC can use Bayesian interim analyses to evaluate the “totality of available evidence” and consider whether the results of a trial would be convincing to skeptical, enthusiastic, and neutral clinical investigators. Decisions to stop a trial early based on the best available data from all relevant trials and on the identification of clinically meaningful differences are most likely to be convincing to patients, clinicians, and investigators.

Center Effects

Although the diversity of NRN NICUs increases the external validity of its studies, large center differences may also obscure effects of interest. Thus, NRN trials are stratified by center whenever possible, with adjustment for center effects in analyses.^58,59 When the sample size is too small to account for centers as fixed effects, or when center-independent predictions are desired,⁶⁰ generalized linear mixed models (GLMM) can include center as a random effect, accounting for both within- and between-center variations.

Hierarchical models such as GLMM have the added benefit of being able to incorporate appropriate center-level data on clinical practice, if available, to help explain and quantify (and not merely account for) center differences.⁶¹

Thus, recently, to evaluate whether differences in initiation of active treatment for extremely premature babies across hospitals may explain the wide center differences in mortality outcomes across the NRN, we used multivariable multilevel logistic regression models to assess clustering of active treatment at the hospital level, by gestational age at birth, after accounting for differences in patient characteristics. Models included infant-level receipt of active treatment as a binary outcome and were adjusted for infant-level characteristics known at birth, and the overall active treatment rate at the hospital level. These models were used to calculate the intra-class correlation for active treatment (i.e., the proportion of variation in active treatment that was attributable to an infant’s hospital of birth) by gestational age at birth.⁶²

Causal Inference from Observational Studies

Clinical trials are the gold standard for developing the evidence base of neonatal medicine. However, randomized trials are often not possible because of ethical considerations, lack of equipoise, or lack of resources. For example, the question of benefit from antenatal steroids for babies born at the edges of viability (25 weeks and under) has not been adequately addressed by a randomized trial, but the ubiquitous use of this treatment in all extremely premature babies does not make a trial viable (ethically or in terms of equipoise) anymore.⁶³ Thus, many open questions in neonatal care can only be addressed by carefully conducted observational studies. Some of the challenges in using observational studies to develop the evidence base for neonatal medicine are briefly outlined below:

Covariates: Selection and control for proper covariates and confounders is essential to proper inference in an observational study. Prospectively planned observational studies in the NRN strive to develop an a priori covariate selection strategy to avoid problems of reverse causation, circular reasoning, and spurious results. The elements of such a strategy typically include the following steps:

1. Principal covariates and confounders are identified from the literature.

2. A primary tier of covariates is identified (typically center, gestational age, gender, exposure to antenatal steroids, and SGA or birth weight) that are always adjusted for, regardless of statistical or clinical significance (so that the effect estimates of interest can be reported as adjusted for such covariates).

3. A secondary tier of covariates is identified that can be adjusted for if indicated by the data or extant literature.

4. Depending on the goals of the analysis (descriptive, hypothesis testing, or prediction), rules for final covariate selection (from the second tier) are set forth a priori based on both statistical and clinical significance.

One of the frequent difficulties faced in neonatal studies is the lack of a universally accepted and comprehensive index measure of baseline level of illness severity. The lack of such a measure can lead to foregone conclusions whereby babies that are sicker at the outset are exposed to more clinical interventions and often have worse outcomes. Different NRN studies have used various measures of baseline level of sickness such as baseline probability of death or NDI (the ongoing NEST trial) or receiving mechanical ventilation for the first 7 days of life.⁶⁴

Causal Inference: Causal inference, whereby we can impart causality and not just association between risk factors (or treatments/interventions) and outcomes from observational studies, is increasingly being used in situations where randomized trials are not possible. Typically, this involves carefully formulating the causal question to be answered, stating the underlying assumptions, using novel analysis methods, and finally using sensitivity analyses to explore the robustness of the conclusions. In the NRN we have used propensity scores modeling^65,66 to compare outcomes for nonrandomized treatments where we use a two-step modeling procedure to reduce bias in such comparisons. For example, in a recent study to examine associations between inhaled nitric oxide (iNO) use and severe BPD or death, we used this approach to even out any imbalance between the iNO treated and untreated groups with regard to baseline risk factors for iNO treatment (to the extent permitted by available data), because iNO treatment was not randomized and there were no other study design features that might have checked such imbalances when iNO treatment was initiated.^67,68

A propensity score was created from a logistic regression model quantifying the probability of receiving iNO treatment, as predicted by a set of well-chosen covariates in the propensity model. Once we obtained the propensity scores, they were ranked and divided into similar groups based on their ranks. Thus, 12 strata were created, each consisting of both iNO treated and control infants that were balanced on key covariates for these two types of infants. A variable representing these strata was included as a predictor in a second logistic regression analysis model to compare the primary outcome (severe BPD or death) between the two treatment groups.

Propensity scores modeling offers a useful conceptual framework to formulate a causal inference question from observational data. In addition, the two-step modeling approach can sometimes circumvent model-fitting and precision problems faced by ordinary regression. However, it is useful to remember that both methods are still vulnerable to the problem of unmeasured or unknown covariates that may be imbalanced across the comparison groups, and can thus produce biased inference.

Longitudinal Analyses

Increased survival of babies at high risk of long-term morbidities emphasizes the need for long-term follow-up. Neurodevelopmental deficiencies (such as IQ, behavioral problems, or conditions such as autism spectrum disorders or attention deficit disorder) may not manifest until later childhood and can only be understood in the context of a child’s overall developmental trajectory. Longitudinal analyses are therefore essential because they are more powerful than cross-sectional studies in terms of explanatory power and statistical efficiency.⁶⁹ For example, the Hypothermia extended follow-up study compared the neurodevelopmental trajectories of treated and control children.⁷⁰ The downside to a longitudinal study are usually higher costs, more attrition and missing data over time, and greater potential for informative missing data at later time points to lead to biased inference.

Multiple Endpoints

In some situations, multiple outcomes may measure different aspects of the same underlying syndrome. We have developed a multivariate approach to model such outcomes simultaneously. This approach conserves degrees of freedom, adjusts for correlations among the outcomes, and avoids multiple comparisons, while allowing for the estimation of outcome-specific effects.^71,72

Biomarkers

Because neonatal research is trending toward longer follow-up to school age, we do not foresee NRN trials using biomarkers as surrogate primary outcomes. However, we do anticipate interest in the validation of prognostic and predictive biomarkers for long-term outcomes for conducting early safety monitoring, counseling parents, identifying candidates for early intervention, and designing studies (as eligibility criteria or stratification factors). Technologies such as MRI and aEEG evaluated in children followed to school age (Hypothermia and SUPPORT trials) and the NRN Biorepository for collection of omics data provide unique opportunities for biomarker identification in extremely premature babies. Biomarker validation methods continue to evolve, as do relevant guidelines such as multistep biomarker development process recommendations by the MicroArray Quality Control-II,⁷³ ACCE evaluation criteria (Analytic and Clinical validity, Clinical utility, Ethical, legal, and social issues),⁷⁴ and recommendations from the Institute of Medicine.⁷⁵ Statistical geneticists in the NRN have contributed to advancements in the field both as coauthors of these guidance documents⁷³ and as developers of innovative methods for the analysis of omics data as illustrated below.

Genetics and Omics Data

The NRN has conducted candidate gene and genome-wide association (GWA) studies of BPD, ROP, ICH, and NEC in extremely premature babies using innovative approaches such as gene/burden-based testing to minimize multiple testing issues and pathways analysis to help interpret the data.^76–78 We have also developed new methods for incorporating covariates into GWA studies with maximal power while minimizing biases.⁷⁷ As whole genome sequencing becomes less expensive it will likely replace GWA studies, with an associated increase in bioinformatics burden.⁷⁹ Future NRN research may also include studies of the microbiome, metagenomics, metabolomics, and proteomics. For example, metagenomics may explore biomarkers for NEC and brain injury in extremely premature and encephalopathic neonates, respectively.

Pharmacokinetic Studies

Complicating factors in PK studies in extremely premature infants include sparse blood samples over time because of infant size, switching of administration mode from IV to oral as an infant matures, and endogenous synthesis of study compound by the infants. We faced each of these three challenges in the NRN Phase II RCT for safety and PK dose-determination of multiple doses of myo-inositol administered every 12 hours for several days (preparatory to a Phase III RCT to prevent ROP). Our solution was to extend and combine population-PK one- and two-compartment models for IV and oral administration into a single model that accounted for change in administration mode from IV to oral and measuring endogenous synthesis of inositol. In addition, our model efficiently combined the sparse repeated measures data from each infant using population-PK methods based on nonlinear mixed effects models.⁸⁰

CONCLUSION

We have discussed some of the specific methodological challenges in the statistical design and analysis of RCTs and observational studies in extremely premature neonates. Although some of these challenges (such as rare outcomes) may not be unique to neonatology, the aggregate constellation of patient population features discussed above is fairly unique and calls for broad understanding and careful consideration of these issues during the conception phase for any rigorous study in this population. As described above, the NRN has developed pragmatic solutions for many of these situations. However, there continue to be unanswered questions and ample room for substantial advances in methodology and new ways of thinking to further refine the design and analysis of neonatal trials and studies and continue to build the evidence base for neonatal medicine.