Assessment of an Updated Neonatal Research Network Extremely Preterm Birth Outcome Model in the Vermont Oxford Network

Matthew A Rysavy; Jeffrey D Horbar; Edward F Bell; Lei Li; Lucy T Greenberg; Jon E Tyson; Ravi M Patel; Waldemar A Carlo; Noelle E Younge; Charles E Green; Erika M Edwards; Susan R Hintz; Michele C Walsh; Jeffrey S Buzas; Abhik Das; Rosemary D Higgins

doi:10.1001/jamapediatrics.2019.6294

. 2020 Mar 2;174(5):e196294. doi: 10.1001/jamapediatrics.2019.6294

Assessment of an Updated Neonatal Research Network Extremely Preterm Birth Outcome Model in the Vermont Oxford Network

Matthew A Rysavy ^1,^✉, Jeffrey D Horbar ^2,³, Edward F Bell ¹, Lei Li ⁴, Lucy T Greenberg ², Jon E Tyson ⁵, Ravi M Patel ⁶, Waldemar A Carlo ⁷, Noelle E Younge ⁸, Charles E Green ⁵, Erika M Edwards ^2,⁹, Susan R Hintz ¹⁰, Michele C Walsh ¹¹, Jeffrey S Buzas ⁹, Abhik Das ¹², Rosemary D Higgins ¹³, for the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network and Vermont Oxford Network

¹Stead Family Department of Pediatrics, University of Iowa, Iowa City

²Vermont Oxford Network, Burlington

³Department of Pediatrics, University of Vermont College of Medicine, Burlington

⁴Biostatistics and Epidemiology Division, RTI International, Research Triangle Park, North Carolina

⁵Center for Clinical Research & Evidence-Based Medicine, University of Texas McGovern Medical School, Houston

⁶Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia

⁷Department of Pediatrics, University of Alabama, Birmingham

⁸Department of Pediatrics, Duke University School of Medicine, Durham, North Carolina

⁹Department of Mathematics and Statistics, College of Engineering and Mathematical Sciences, University of Vermont, Burlington

¹⁰Department of Pediatrics, Stanford University School of Medicine, Palo Alto, California

¹¹Department of Pediatrics, Case Western Reserve University, Cleveland, Ohio

¹²Biostatistics and Epidemiology Division, RTI International, Rockville, Maryland

¹³Office of Research, George Mason University College of Health and Human Services, Fairfax, Virginia

Group Information: The Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network and Vermont Oxford Network members appear at the end of the article.

Accepted for Publication: December 12, 2019

^✉

Corresponding Author: Matthew A. Rysavy, MD, PhD, Stead Family Department of Pediatrics, University of Iowa, 200 Hawkins Dr, Iowa City, IA 52242 (matthew-rysavy@uiowa.edu).

Published Online: March 2, 2020. doi:10.1001/jamapediatrics.2019.6294

Author Contributions: Dr Li and Ms Greenberg had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Rysavy, Horbar, Bell, Tyson, Carlo, Younge, Walsh, Higgins.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Rysavy, Li.

Critical revision of the manuscript for important intellectual content: Rysavy, Horbar, Bell, Greenberg, Tyson, Patel, Carlo, Younge, Green, Edwards, Hintz, Walsh, Buzas, Das, Higgins.

Statistical analysis: Li, Greenberg, Green, Edwards, Walsh, Buzas, Das.

Obtained funding: Bell, Carlo, Walsh.

Administrative, technical, or material support: Bell, Patel, Walsh, Higgins.

Supervision: Horbar, Bell, Tyson, Carlo, Hintz, Walsh, Higgins.

Conflict of Interest Disclosures: Dr Rysavy reported receiving grants from the National Institutes of Health (NIH) and travel funds from the Vermont Oxford Network (VON) during the conduct of the study. Dr Horbar reported receiving personal fees from VON outside the submitted work and serving as an unpaid member of the Board of Directors of VON. Dr Bell reported receiving grants from the NIH. Dr Tyson reported receiving grants from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) during the conduct of the study and grants from the NICHD outside the submitted work. Dr Patel reported receiving grants from the NIH during the conduct of the study. Dr Carlo served as a paid member of the board of directors of Mednax and received grants from the NIH during the study. Dr Younge reported receiving grants from the NIH during the conduct of the study. Dr Green reported receiving grants from the NICHD during the conduct of the study. Dr Edwards reported receiving grants from VON during the conduct of the study. Dr Hintz reported receiving grants from the NICHD during the conduct of the study. Dr Walsh reported receiving grants from the NIH. Dr Das reported receiving grants from the NICHD and the NIH during the conduct of the study. No other disclosures were reported.

Funding/Support: This study was supported by grants F32 HD098782 (Dr Rysavy), UG1 HD027904, UG1 HD021364 (Dr Walsh), UG1 HD027853, UG1 HD040492, UG1 HD027851, UG1 HD027856, UG1 HD021373, UG1 HD068278, U24 HD095254 (Dr Das), UG1 HD027880, UG1 HD053119, UG1 HD034216 (Dr Carlo), UG1 HD053109 (Dr Bell), UG1 HD053089, UG1 HD068244, UG1 HD068263, UG1 HD040689, UG1 HD053124, UG1 HD021385, and UG1 HD027871 from the NIH and the NICHD, which supported for the Neonatal Research Network’s (NRN’s) Generic Database and Follow-up Study and VON through cooperative agreements.

Role of the Funder/Sponsor: The NICHD staff had input into the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.

Group Members: The following investigators with the NICHD NRN, in addition to those listed as authors, participated in this study and are listed by center. NRN Steering Committee Chair: Michael S. Caplan, MD: University of Chicago, Pritzker School of Medicine (2006-2011); Richard A. Polin, MD: Division of Neonatology, College of Physicians and Surgeons, Columbia University (2011-present). Alpert Medical School of Brown University and Women & Infants Hospital of Rhode Island: Abbot R. Laptook, MD; Angelita M. Hensman, PhD, RNC-NIC; William Oh, MD; Kristin M. Basso, MaT, RN; Elisa Vieira, BSN, RN; Martin Keszler, MD; Betty R. Vohr, MD; Elisabeth C. McGowan, MD; Robert Burke, MD; Melinda Caskey, MD; Katharine Johnson, MD; Barbara Alksninis, PNP; Mary Lenore Keszler, MD; Theresa M. Leach, Med, CAES; Bonnie E. Stephens, MD; Victoria E. Watson, MS, CAS; Suzy Ventura; Andrea M. Knoll. Case Western Reserve University, Rainbow Babies & Children's Hospital: Anna Marie Hibbs, MD; Deanne E. Wilson-Costello, MD; Nancy S. Newman, BA, RN; Bonnie S. Siner, RN; Monika Bhola, MD; Gulgun Yalcinkaya, MD; Harriet G. Friedman, MA. Cincinnati Children's Hospital Medical Center, University Hospital, and Good Samaritan Hospital: Kurt Schibler, MD; Kate Bridges, MD; Barbara Alexander, RN; Cathy Grisby, BSN, CCRC; Jody Hessling, RN; Estelle E. Fischer, MHSA MBA; Lenora D. Jackson, CRC; Kristin Kirker, CRC; Holly L. Mincey, RN, BSN; Greg Muthig, BS; Teresa L. Gratton, PA; Jean J. Steichen, MD; Kimberly Yolton, PhD. Duke University School of Medicine, University Hospital, University of North Carolina, and Duke Regional Hospital: C. Michael Cotten, MD, MHS; Ronald N. Goldberg, MD; Ricki F. Goldstein, MD; Kimberley A. Fisher, PhD, FNP-BC, IBCLC; Kathy J. Auten, MSHS; Katherine A. Foy, RN; Sandra Grimes, RN, BSN; Joanne Finkle, RN, JD; Kathryn E. Gustafson, PhD; Melody B. Lohmeyer, RN, MSN; Matthew M. Laughon, MD MPH; Carl L. Bose, MD; Janice Bernhardt, MS RN; Gennie Bose, RN; Cindy Clark, RN. Emory University, Children’s Healthcare of Atlanta, Grady Memorial Hospital: David P. Carlton, MD; Barbara J. Stoll, MD; Ellen C. Hale, RN, BS, CCRC; Yvonne C. Loggins, RN, BSN; Ira Adams-Chapman, MD; Maureen Mulligan LaRossa, RN; Sheena L. Carter, PhD. NICHD: Stephanie Wilson Archer, MA. Indiana University, University Hospital, Methodist Hospital, Riley Hospital for Children, and Eskenazi Health: Gregory M. Sokol, MD; Brenda B. Poindexter, MD, MS; Anna Maria Dusick, MD (deceased); Dianne E. Herron, RN, CCRC; Abbey C. Hines, PsyD; Carolyn Lytle, MD, MPH; Lucy C. Miller, RN, BSN, CCRC; Heike M. Minnich, PsyD, HSPP; Lu Ann Papile, MD; Leslie Dawn Wilson, BSN, CCRC. McGovern Medical School at The University of Texas Health Science Center at Houston, Children's Memorial Hermann Hospital, and Lyndon Baines Johnson General Hospital/Harris County Hospital District: Kathleen A. Kennedy, MD, MPH; Nora I. Alaniz, BS; Katrina Burson, RN, BSN; Patricia W. Evans, MD; Beverly Foley Harris, RN, BSN; Margarita Jiminez, MD, MPH; Anna E. Lis, RN, BSN; Sarah Martin, RN, BSN; Georgia E. McDavid, RN; Brenda H. Morris, MD; M. Layne Poundstone, RN, BSN; Peggy Robichaux, RN, BSN; Saba Siddiki, MD; Maegan C. Simmons, RN; Patti L. Pierce Tate, RCP; Sharon L. Wright, MT (ASCP). Nationwide Children’s Hospital and the Ohio State University Medical Center: Pablo J. Sánchez, MD; Leif D. Nelin, MD; Sudarshan R. Jadcherla, MD; Patricia Luzader, RN; Christine A. Fortney, PhD, RN; Gail E. Besner; Nehal A. Parikh, MD. RTI International: Marie G. Gantz, PhD; W. Kenneth Poole, PhD (deceased); Dennis Wallace, PhD; Jamie E. Newman, PhD, MPH; Jeanette O’Donnell Auman, BS; Margaret M. Crawford, BS, CCRP; Carolyn M. Petrie Huitema, MS, CCRP; Kristin M. Zaterka-Baxter, RN, BSN, CCRP. Stanford University, Dominican Hospital, El Camino Hospital, and Lucile Packard Children's Hospital: Krisa P. Van Meurs, MD; David K. Stevenson, MD; Marian M. Adams, MD; M. Bethany Ball, BSc, CCRC; Andrew W. Palmquist, RN, BSN; Melinda S. Proud, RCP; Barbara Bentley, PsychD, MSEd; Elizabeth Bruno, PhD; Maria Elena DeAnda, PhD; Anne M. DeBattista, RN, PNP; Beth Earhart, PhD; Jean G. Kohn, MD, MPH; Casey E. Krueger, PhD; Renee P. Pyle, PhD; Heather Taylor, PhD; Hali E. Weiss, MD. Tufts Medical Center, Floating Hospital for Children: Ivan D. Frantz III, MD; John M. Fiascone, MD; Brenda L. MacKinnon, RNC; Ellen Nylen, RN, BSN; Anne Furey, MPH; Elisabeth C. McGowan, MD; Cecelia E. Sibley, PT, MHA; Ana K. Brussa, MS, OTR/L. University of Alabama at Birmingham Health System and Children’s Hospital of Alabama: Namasivayam Ambalavanan, MD; Myriam Peralta-Carcelen, MD, MPH; Monica V. Collins, RN, BSN, MaEd; Shirley S. Cosby, RN, BSN; Fred J. Biasini, PhD; Kathleen G. Nelson, MD; Cryshelle S. Patterson, PhD; Vivien A. Phillips, RN, BSN; Sally Whitley, MA, OTR-L, FAOTA; Carin Kiser, MD; Leigh Ann Smith, CRNP; Richard V. Rector, PhD; Sarah Ryan, PhD; Kristy Domnanovich, PhD; Leslie Rodrigues, PhD; Sheree Chapman York, PT, DPT, MS, PCS. University of Iowa and Mercy Medical Center: Jane E. Brumbaugh, MD; Tarah T. Colaizy, MD, MPH; Dan L. Ellsbury, MD; John A. Widness, MD; Michael J. Acarregui, MD, MBA; Karen J. Johnson, RN, BSN; Donia B. Campbell, RNC-NIC; Diane L. Eastman, RN, CPNP, MA. University of New Mexico Health Sciences Center: Kristi L. Watterberg, MD; Robin K. Ohls, MD; Janell F. Fuller, MD; Conra Backstrom Lacy, RN; Rebecca A. Montman, BSN, RNC; Jean R. Lowe, PhD; Andrea Freeman Duncan, MD; Sandra Brown, BSN; Carol Hartenberger, BSN, MPH. University of Pennsylvania, Hospital of the University of Pennsylvania, Pennsylvania Hospital, and Children's Hospital of Philadelphia: Barbara Schmidt, MD, MSc; Haresh Kirpalani, MB, MSc; Sara B. DeMauro, MD, MSCE; Aasma S. Chaudhary, BS, RRT; Soraya Abbasi, MD; Toni Mancini, RN, BSN, CCRC; Dara Cucinotta. University of Rochester Medical Center, Golisano Children's Hospital, and the University at Buffalo Women's and Children's Hospital of Buffalo: Dale L. Phelps, MD; Gary J. Myers, MD; Linda J. Reubens, RN, CCRC; Erica Burnell, RN; Diane Hust, MS, RN, CS; Julie Babish Johnson, MSW; Rosemary L. Jensen; Emily Kushner, MA; Joan Merzbach, LMSW; Kelley Yost, PhD; Lauren Zwetsch, RN, MS, PNP; Satyan Lakshminrusimha, MD; Anne Marie Reynolds, MD, MPH; Osman Farooq, MD; Ashley Williams, MS, Ed; William A. Zorn, PhD. University of Texas Southwestern Medical Center at Dallas, Parkland Health & Hospital System, and Children's Medical Center Dallas: Pablo J. Sánchez, MD; Roy J. Heyne, MD; Walid A. Salhab, MD; Charles R. Rosenfeld, MD; Luc P. Brion, MD; LiJun Chen, RN, PhD; Alicia Guzman; Melissa H. Leps, RN; Nancy A. Miller, RN; Diana M. Vasil, MSN, BSN, RNC, NIC; Lizette E. Torres, RN; Gaynelle Hensley, RN; Sally S. Adams, MS, RN, CPNP; Linda A. Madden, RN CPNP; Elizabeth Heyne, PsyD PA-C; Janet S. Morgan, RN; Catherine Twell Boatman, MS, CIMI. University of Utah University Hospital, Intermountain Medical Center, LDS Hospital, and Primary Children's Medical: Roger G. Faix, MD; Bradley A. Yoder, MD; Karen A. Osborne, RN BSN CCRC; Shawna Baker, RN; Karie Bird, RN, BSN; Jill Burnett, RNC, BSN; Jennifer J. Jensen, RN, BSN; Cynthia Spencer, RNC, BSN; Mike Steffen, PhD; Kimberlee Weaver-Lewis, RN, MS; Sarah Winter, MD; Karen Zanetti, RN. Wayne State University, University of Michigan, Hutzel Women’s Hospital, Sinai-Grace Hospital, and Children’s Hospital of Michigan: Seetha Shankaran, MD; Sanjay Chawla, MD, Girija Natarajan, MD, Monika Bajaj, MD, Beena Sood, MD, Athina Pappas, MD; John Barks, MD; Rebecca Bara, RN, BSN; Laura A. Goldston, MA; Mary Johnson, RN, BSN; Laura Sumner, RN, BSN; Kara Sawaya, RN, BSN; Kathleen Weingarden, RN, BSN; Mary Christensen, RT; Stephanie Wiggins, MS; Daine F. White, RN; Eunice Woldt, RN, MSN. Yale University, Yale-New Haven Children’s Hospital, and Bridgeport Hospital: Richard A. Ehrenkranz, MD; Harris Jacobs, MD; Christine G. Butler, MD; Patricia Cervone, RN; Sheila Greisman, RN; Monica Konstantino, RN, BSN; JoAnn Poulsen, RN; Janet Taft, RN, BSN; Joanne Williams, RN, BSN; Elaine Romano, MSN.

Disclaimer: The comments and views of the authors do not necessarily represent the views of NICHD, the NIH, the Department of Health and Human Services, or the US government.

^✉

Corresponding author.

PMCID: PMC7052789 PMID: 32119065

This prognostic study assesses the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network extremely preterm infant outcome model.

Key Points

Question

Do differences in infant survival among hospitals and over time affect a prognostic model widely used in extremely preterm birth counseling?

Findings

In this prognostic study of most actively treated extremely preterm infants in the United States in 2006 to 2012 and 2013 to 2016, survival increased from 66% to 70%, and model prediction was moderate. The birth hospital and gestational age contributed equally to prediction of survival .

Meaning

For extremely preterm birth, an area of medicine with substantial variation among hospitals and changing outcomes, prognostic models used in clinical practice may require accounting for local outcomes and periodic updating to remain relevant.

Abstract

Importance

The Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network (NRN) extremely preterm birth outcome model is widely used for prognostication by practitioners caring for families expecting extremely preterm birth. The model provides information on mean outcomes from 1998 to 2003 and does not account for substantial variation in outcomes among US hospitals.

Objective

To update and validate the NRN extremely preterm birth outcome model for most extremely preterm infants in the United States.

Design, Setting, and Participants

This prognostic study included 3 observational cohorts from January 1, 2006, to December 31, 2016, at 19 US centers in the NRN (derivation cohort) and 637 US centers in Vermont Oxford Network (VON) (validation cohorts). Actively treated infants born at 22 weeks’ 0 days’ to 25 weeks’ 6 days’ gestation and weighing 401 to 1000 g, including 4176 in the NRN for 2006 to 2012, 45 179 in VON for 2006 to 2012, and 25 969 in VON for 2013 to 2016, were studied. VON cohorts comprised more than 85% of eligible US births. Data analysis was performed from May 1, 2017, to March 31, 2019.

Exposures

Predictive variables used in the original model, including infant sex, birth weight, plurality, gestational age at birth, and exposure to antenatal corticosteroids.

Main Outcomes and Measures

The main outcome was death before discharge. Secondary outcomes included neurodevelopmental impairment at 18 to 26 months’ corrected age and measures of hospital resource use (days of hospitalization and ventilator use).

Results

Among 4176 actively treated infants in the NRN cohort (48% female; mean [SD] gestational age, 24.2 [0.8] weeks), survival was 63% vs 62% among 3702 infants in the era of the original model (47% female; mean [SD] gestational age, 24.2 [0.8] weeks). In the concurrent (2006-2012) VON cohort, survival was 66% among 45 179 actively treated infants (47% female; mean [SD] gestational age, 24.1 [0.8] weeks) and 70% among 25 969 infants from 2013 to 2016 (48% female; mean [SD] gestational age, 24.1 [0.8] weeks). Model C statistics were 0.74 in the 2006-2012 validation cohort and 0.73 in the 2013-2016 validation cohort. With the use of decision curve analysis to compare the model with a gestational age–only approach to prognostication, the updated model showed a predictive advantage. The birth hospital contributed equally as much to prediction of survival as gestational age (20%) but less than the other factors combined (60%).

Conclusions and Relevance

An updated model using well-known factors to predict survival for extremely preterm infants performed moderately well when applied to large US cohorts. Because survival rates change over time, the model requires periodic updating. The hospital of birth contributed substantially to outcome prediction.

Introduction

Accurate estimates of prognosis after extremely preterm birth allow practitioners and families to plan for potential outcomes and may guide clinical decision-making.^1,2 Since 2008, the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Neonatal Research Network (NRN) has published on its website data describing outcomes of infants born to assist families and practitioners managing extremely preterm birth.³ The website displays outcomes from a model developed using data for infants born at NRN hospitals from 1998 to 2003. The model takes into account 5 factors: an infant’s sex, birth weight, plurality (singleton vs multiple gestation), gestational age at birth, and exposure to antenatal corticosteroids. Prognostic estimates that accounted for these 5 factors were more accurate than those based on gestational age alone.⁴

The NICHD NRN extremely preterm birth outcome model is widely used to inform perinatal prognostic discussions⁵ and is cited in guidelines of professional societies.^6,7 However, interventions and outcomes for extremely preterm infants have changed since its development.^8,9 In addition, studies^10,11,12,13 have described substantial variation in extremely preterm infant outcomes among US hospitals, including within the NRN.

The aim of this investigation was to update the NICHD NRN extremely preterm birth outcome model to reflect more recent neonatal practices and outcomes. We evaluated the importance of hospital differences in outcomes relative to the other factors included in the model. In addition, we evaluated the updated model using data for most US extremely preterm births from 2006 to 2012 and 2013 to 2016 to assess its performance over time.

Methods

In this prognostic study, we included data for inborn infants born at 22 weeks’ 0 days’ to 25 weeks’ 6 days’ gestation and weighing 401 to 1000 g. Infants with major anomalies, with birth weights greater than the 97th percentile for gestational age (raising the possibility that gestational age may have been underestimated^14,15), or who were not born at participating hospitals were excluded. These same criteria were used in the development of the original model.⁴ The institutional review boards at each NRN site approved data collection protocols. Waiver of consent was granted by the institutional review boards at 17 of 19 centers for the collection of in-hospital data and 4 of 19 centers for collection of follow-up data. All data were deidentified. The University of Vermont Institutional Review Board considered this research using the Vermont Oxford Network (VON) deidentified data repository to be not human subjects research. Model derivation, validation, and reporting were conducted according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.¹⁶

Derivation Cohort

The new model was derived using data for infants born from April 1, 2006, to December 31, 2012, at 19 centers participating in the NRN. Because outcome data for infants who did not receive any potentially life-sustaining treatment cannot be used to estimate the probability of survival with intensive care,¹⁷ the model was derived for infants who received any active treatment after birth. Active treatment was defined as surfactant therapy, endotracheal intubation, ventilatory support (including continuous positive airway pressure, bag-valve-mask ventilation, or mechanical ventilation), epinephrine, or chest compressions.¹⁸ This change from the original model, which defined active treatment based solely on administration of mechanical ventilation, reflects changes in practice in the intervening period.⁴

Validation Cohorts

The updated model was applied to births at 637 US hospitals in the VON database from January 1, 2006, to December 31, 2012, using the derivation cohort inclusion and exclusion criteria. VON included more than 85% of all US infants born at 22 to 25 weeks’ gestation and weighing 400 to 1000 g during the study period.¹⁹ Inclusion was not limited by the level or volume of the birth hospital. Although NRN data were derived from hospitals with level III and IV neonatal intensive care units (NICUs), the VON data included infants born at 79 hospitals with level I NICUs (restrictions on assisted ventilatory support), 176 hospitals with level II NICUs (no ventilatory support restrictions and no neonatal surgery), and 382 level III or IV NICUs (no ventilatory support restrictions and perform neonatal surgery). Many infants were transferred to other hospitals during their clinical care. Hospitals participating in both the NRN and VON were excluded from the validation data set for the overlapping years. A second cohort from the VON database comprising infants born from January 1, 2013, to December 31, 2016, from the same hospitals and with the same inclusion and exclusion criteria was used to test the stability of hospital-specific observed and estimated outcomes over time.

Predictive Variables

Infant data were collected at each hospital using standardized protocols. Gestational age at birth was defined in completed weeks (eg, 23 weeks, representing births at 23 weeks 0 days through 23 weeks 6 days) and determined using standard protocols that prioritized the best obstetric estimate.²⁰ Birth weight was measured in grams and compared with the gestational age– and sex-specific growth curves used in the development of the original model.^4,15 Plurality was defined as singleton birth or multiple birth. Exposure to antenatal corticosteroids was defined as a binary variable with exposure including any antenatal receipt of betamethasone or dexamethasone for acceleration of fetal maturation regardless of duration, dose, or timing relative to birth.

Outcomes

Survival was measured as survival to discharge or to 1 year of age if still hospitalized. Data on in-hospital resource use and neurodevelopmental outcomes at 18 to 26 months’ corrected age were available for the NRN derivation cohort and are described in eTable 1 and eTable 2 in the Supplement.

Resource use was described in infant-days of mechanical ventilation per surviving infant and infant-days of hospitalization per surviving infant. Neurodevelopmental outcomes were ascertained by standardized, annually certified examiners who were not blinded to infants’ medical information. They were classified based on cognitive development assessed with the Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III), gross motor function as assessed by the Gross Motor Function Classification System (GMFCS), and the presence of cerebral palsy and hearing and vision impairment. Profound neurodevelopmental impairment was defined as a BSID-III cognitive score that was untestable or a GMFCS level of 5 (on a scale of 0 [normal] to 5 [most impaired]). Moderate-severe neurodevelopmental impairment was defined as a BSID-III cognitive score of less than 85 (ie, >1 SD below the scale mean; mean [SD], 100 [15]), moderate or severe cerebral palsy, a GMFCS level of 2 to 5, bilateral blindness, or severe hearing impairment that could not be corrected with bilateral amplification.

Statistical Analysis

We compared descriptive characteristics of infants in the derivation and validation cohorts with the characteristics of infants in the cohort used to develop the original model (1998-2003).⁴ Analysis of the NRN data (L.L.) was conducted using SAS Enterprise Guide, version 7.15 (SAS Institute Inc) and Stata/MP, version 16.0 (StataCorp). Analysis of VON data (L.T.G.) was conducted using R, version 3.3.2 (R Foundation for Statistical Computing). Using the NRN data set, we derived a multivariable logistic mixed model with survival as the dependent outcome; birth weight, gestational age, plurality, and antenatal corticosteroid exposure as independent variables; and hospital of birth as a random intercept. These same factors were included in the original model.⁴ Neurodevelopmental outcomes and resource use were described for actively treated infants stratified by their probability of survival.

For external validation, the updated model was applied to eligible infants in nonoverlapping hospitals in the 2006 to 2012 VON cohort. We estimated random intercepts for all VON hospitals using k-fold cross-validation to evaluate model fit. Modeling discrimination was assessed using the C statistic. For the outcome of survival, the C statistic represents the probability that for any randomly selected pair of individuals in which one lives and the other dies, the model estimates a higher probability for the one who lives.^21,22 Model calibration was assessed by comparing estimated and observed outcomes for the whole sample across several subgroups of infant characteristics.

To complement discrimination and calibration metrics, we also performed a decision curve analysis that compared the predicted outcomes of treating all or no patients by gestational age—the approach recommended in some guidelines²³—with treating a targeted group based on model-estimated survival. The decision curve analysis presents various predicted survival rates above which infants might be selected to receive active treatment (ie, threshold probabilities) and presents the predictive net benefit of using that threshold. The predictive net benefit was calculated as follows: [proportion of treated infants who lived] – [(proportion of treated infants who died) × (threshold probability)/(1-threshold probability)].^24,25

To evaluate the importance of hospital differences in outcomes relative to the other factors included in the model, we compared observed and model-estimated outcomes with and without accounting for the hospital-specific intercept term in the estimate. We calculated the relative contribution of model variables to outcome estimation using the differences between the log-likelihood value of the full model and of models without each variable.²⁶ To assess the stability of hospital-specific infant outcome estimates over time, the hospital-specific model with intercept terms derived in 2006 to 2012 was applied to the 2013 to 2016 cohort and observed and estimated outcomes were compared.

As a sensitivity analysis, we also compared the outcomes estimated when the model was applied to all infants born in the validation cohorts regardless of whether they were actively treated, as previously performed.⁴ The results of this analysis estimated the maximum potential survival that might result if all infants in the cohorts were actively treated.⁴ Data analysis was performed from May 1, 2017, to March 31, 2019.

Results

Table 1. Characteristics of Actively Treated Infants in Each Cohort^a.

Characteristic	NRN		VON
Characteristic	1998-2003 (n = 3702)	2006-2012 (n = 4176)	2006-2012 (n = 45 179)	2013-2016 (n = 25 969)
Prenatal care	93	95	95	96
Cesarean delivery	48	58	64	65
Antenatal corticosteroids	80	85	79	86
Race/ethnicity
Black^b	45	38	36	36
White^b	36	39	40	38
Hispanic^b	17	15	19	20
Singleton	76	74	76	76
Female	47	48	47	48
Gestational age, mean (SD), wk	24.2 (0.8)	24.2 (0.8)	24.1 (0.8)	24.1 (0.8)
Gestational age, wk
22	3	2	3	3
23	18	18	20	21
24	38	37	36	36
25	42	42	41	40
Birth weight, mean (SD), g	670 (118)	676 (125)	671 (125)	671 (128)
Apgar score ≤3
1 min	50	57	50	52
5 min	15	22	17	19
Length of stay, median (5th-95th percentile), d^c	88 (0-177)	96 (1-197)	95 (1-188)	102 (2-204)
Neonatal level of care at hospital of birth^d
Level III or IV NICU	100	100	81	80
Level II NICU	0	0	17	18
Level I NICU	0	0	2	2

Open in a new tab

Abbreviations: NICU, neonatal intensive care unit; NRN, Neonatal Research Network; VON, Vermont Oxford Network.

^{^a}

Data are presented as percentage of infants unless otherwise indicated.

^{^b}

Maternal race/ethnicity determined by self-report as obtained by interview or, if unavailable, birth certificate or medical records.

^{^c}

In the 2006 to 2012 NRN cohort, 31 infants (0.7%) were still hospitalized at 1 year. In the 2006-2012 VON cohort, 241 infants (0.5%) were still hospitalized at 1 year. In the 2013 to 2016 VON cohort, 169 infants (0.6%) were still hospitalized at 1 year. Data were unavailable for the 1998-2003 NRN cohort.

^{^d}

The NICU designations include level I (restrictions on assisted ventilatory support), level II (no ventilatory support restrictions and no neonatal surgery), level III (no ventilatory support restrictions and perform neonatal surgery except cardiac surgery that requires bypass), and level IV (no ventilatory support restrictions and perform neonatal surgery, including cardiac surgery that required bypass). Because NICU designations can change over time and are assessed infrequently in the data sets, the proportions reported in the Table represent the designations most recently available at the end of each study period.

Model Derivation

Odds ratios for each variable included in the updated and original 5-factor models are shown in Figure 1. Except for the increase in the point estimate for survival for infants born at 22 weeks’ compared with 25 weeks’ gestation, there was little change in other model coefficients.

Model-estimated survival outcomes for infants in the 2006 to 2012 NRN and VON validation cohorts by gestational age at birth are shown in eFigure 1 in the Supplement. The mean (SD) estimated probability of survival for actively treated infants born at 22 weeks’ gestation in the NRN was 25% (11%). However, there was considerable variation in the probability of survival for infants born during each gestational age week, with a range of 8% to 61% based on the other factors in the model. Likewise, the mean probability of survival for actively treated infants born at 23 weeks’ gestation in the NRN was 39% (14%), with a range of 9% to 76%.

When accounting for the resources used for infants who died, for each infant who lived with an estimated survival probability of 11% to 20%, 153 infant-days of mechanical ventilation and 280 infant-days of hospitalization were used (eTable 1 in the Supplement). For each infant with a probability of survival greater than 80%, 28 infant-days of mechanical ventilation and 115 infant-days of hospitalization were used per infant who lived.

Of 4176 actively treated infants in the NRN cohort, 3927 (94%) had follow-up outcomes known at 18 to 26 months’ corrected age (eTable 2 in the Supplement). Rates of profound neurodevelopmental impairment were 1% among infants with an estimated survival probability greater than 80% and 11% among infants with an estimated survival probability of 11% to 20%. Rates of moderate-severe neurodevelopmental impairment (including profound impairment) were 21% among infants with an estimated survival probability greater than 80% and 78% among infants with an estimated survival probability of 11% to 20%. No actively treated infants with a probability of survival of 10% or less (n = 6) lived to follow-up.

Model Validation

Estimates of model discrimination and calibration when the model was applied to the 2006 to 2012 VON cohort are given in Table 2. The original 5-factor model for mortality had a C statistic of 0.75 in the previously published derivation cohort of NRN hospitals (1998-2003)⁴; discrimination was similar in the 2006 to 2012 VON cohort (C statistic, 0.73). By comparison, the updated model incorporating hospital-specific intercepts had a C statistic of 0.74 in the 2006 to 2012 VON cohort.

Table 2. Comparison of the Updated and Original 5-Factor Models in the VON 2006-2012 Cohort.

Variable	Gestational Age Only	Original Model	Updated Model
Variable	Gestational Age Only	Original Model	Without Center Terms	With Center Terms
Mean absolute difference, %^a	8.5	5.5	3.5	1.6
Range of observed minus expected outcomes, %^a	−18.7 to 18.2	−1.7 to 11.5	−1.7 to 7.8	−3.7 to 3.4
C statistic (95% CI)	0.679 (0.674 to 0.684)	0.727 (0.722 to 0.732)	0.728 (0.723 to 0.733)	0.744 (0.739 to 0.749)

Open in a new tab

Abbreviation: VON, Vermont Oxford Network.

^{^a}

The range of values for observed minus expected percent differences are for 24 subgroup combinations of the 5 risk factors.

Compared with the original model and the use of gestational age mean outcomes, the updated model that accounted for hospital of birth was better calibrated to recent outcomes (Figure 2). When model-estimated and observed outcomes were compared, the calibration slope was 0.99 and the intercept was 0.03 (eFigure 2 in the Supplement). Decision curve analysis demonstrated that the updated model provided greater predictive net benefit vs the original model for a wide range of decision thresholds (eFigure 3 in the Supplement). The model that included hospital of birth outperformed both the original model and the updated model that did not account for hospital of birth. This model found a greater predictive net benefit at all gestational ages across a range of threshold probabilities. The predictive net benefit of the updated model was most pronounced at 22 and 23 weeks’ gestation compared with a gestational age–based approach.

Figure 2. — Observed and model-predicted survival rates in the VON 2006-2012 cohort were compared for infants grouped by birth weight quartile for gestational age (<25%, 25%-75%, or >75%), sex (male or female), plurality (singleton or multiple gestation), and exposure to antenatal steroids (yes or no) as presented in the development of the original model.⁴

Hospital of Birth

As indicated in Table 3, hospital of birth contributed to model estimation of survival equally as much as gestational age (20%). Factors besides hospital of birth and gestational age contributed 60% of model predictive ability. When the model without hospital of birth was applied to individual hospitals in the 2006 to 2012 VON cohort with more than 20 infants during the study period, 125 of 386 (32%) hospitals had an observed hospital survival rate significantly different from the estimated rate (eFigure 4 in the Supplement), suggesting that this model did not adequately account for variation in outcomes across hospitals.

Table 3. Relative Contribution of Variables to the Multivariable Model^a.

Variable	Decrease in Log Likelihood	Relative Contribution, %
Birth weight	79.4	36
Infant sex	31.3	14
Antenatal corticosteroids	18.9	9
Plurality	3.2	1
Gestational age	45.0	20
Hospital of birth	43.7	20

Open in a new tab

^{^a}

Values represent the relative contribution of each variable (as a proportion of the total change in model log-likelihood value) to model predictive ability. We estimated the relative contribution of individual variables to the overall predictive value by using the differences between the log likelihood of the full model and the log likelihood of a model without each variable. Relative contribution was defined as the ratio of the variable’s log-likelihood difference to the sum of the model’s 6 log-likelihood differences multiplied by 100.

To test whether hospital-specific terms established for the 2006 to 2012 VON cohort could be used prospectively for the same hospitals, the NRN model applied to VON births from 2006 to 2012 was also applied to VON births from 2013 to 2016. For the same 386 VON hospitals, 34 (9%) had observed outcomes significantly different from predicted outcomes (eFigure 5 in the Supplement), with 30 of 34 hospitals having better than predicted outcomes, suggesting an improvement in outcomes over time not accounted for by the model. The C statistic of the model in the 2013 to 2016 cohort was 0.73.

Sensitivity Analyses

The predicted rate of survival among all infants born in study hospitals, assuming all infants were actively treated (ie, the maximum potential rate), approximated the observed and predicted survival rates among infants who were actively treated (eTable 3 in the Supplement). Of note, rates of both active treatment and infant survival increased from the 2006 to 2012 period to the 2013 to 2016 period. Predicted survival rates also increased during the period in association with increased use of antenatal corticosteroids (Table 1).

Discussion

The update to the NICHD NRN extremely preterm birth outcome model demonstrated moderate discrimination and calibration when applied to most actively treated extremely preterm infants in the United States. Compared with gestational age–only approaches, the model had a predictive advantage by decision curve analysis. The incorporation of hospital of birth into model estimates contributed substantially to outcome estimation. Although the association of the 5 infant characteristics in the model with infant outcomes remained relatively stable from the original era to the most recent one, outcomes improved during the 2 periods studied, suggesting a need to periodically recalibrate the model, including for hospital-specific outcomes.

Because of variation in hospital outcomes, some practitioners have advocated for using only local statistics for predicting infant survival; however, given the small number of extremely preterm infants born at any given hospital, this approach may result in imprecise estimates and may preclude taking into account important prognostic variables, such as infant sex or antenatal corticosteroid exposure.^7,27 In our study, approximately 60% of model estimation of survival resulted from variables besides hospital of birth and gestational age.

Our approach accounted for hospital-specific outcomes while also benefiting from large data sets for deriving and validating the prognostic value of several important infant variables. The contribution of the birth hospital to model prediction may represent differences in practices or case mix among hospitals not accounted for by other model variables. Because prognostic estimates and treatment decisions are made within individual hospitals, our model has been calibrated to individual hospitals as recommended by several methodologists.^28,29,30 Our approach assumes that past hospital-specific neonatal survival rates reliably predict future survival rates, as previously shown in a cohort of US hospitals.³¹ Our data suggest that outcomes of extremely preterm infants at many US hospitals vary substantially from the mean.

The updated model assigned more accurate probabilities of infant outcomes than the original model when applied to births in recent large US cohorts (Figure 2 and eFigure 2 in the Supplement). However, the improved model calibration should be distinguished from model discrimination as represented by the C statistic, a measure of the ability of the model to separate individuals who develop the outcome of interest from those who do not. The C statistic of the updated model was not substantially different from that observed when deriving the original model⁴ and in independent validations of that model^32,33 and demonstrated the continued difficulty of estimating outcomes for extremely preterm infants at the time of birth. Historically, signs from physical examination, such as fusion of the eyelids, were used to determine viability at extremely preterm gestations, although their utility has not been borne out.^34,35 The use of the combined 5 factors in the model was again found to outperform gestational age alone as a tool for predicting survival,⁴ and the predictive benefits of a model-based vs gestational age–based approach were demonstrated in the decision curve analysis. In the future, other methods, such as the use of biochemical markers of infant maturity, may enhance our ability to estimate infant outcomes.³⁶

Although we attempted to maintain the same form as the original 5-factor model, which has been in use for more than a decade and is familiar to many practitioners, the updated model differs from the original in key ways. The study deriving the original model used mechanical ventilation that required endotracheal intubation as a marker of active treatment for infants born in 1999 to 2003. We defined active treatment using a broader definition that accounted for changes in the use of noninvasive ventilation³⁷; 2.5% of 2394 infants in the derivation cohort did not require endotracheal intubation and mechanical ventilation. We also defined survival as occurring at discharge rather than at follow-up, the time point used in the original model, to validate the model in the VON cohorts, which included data through hospital discharge. These changes may explain the small difference in survival seen between the original 1998 to 2003 cohort and the 2006 to 2012 NRN cohort included in this study. In addition, information on timing of antenatal corticosteroid administration was not available for the cohorts used to update the model; the original model included antenatal corticosteroid exposure only if taking place within 7 days of birth.

Limitations

The updated model has important limitations, many of which are shared by the original model. First, the model includes birth weight, which can be estimated only imprecisely before birth.³⁸ Second, the best obstetric estimates of gestational age for most pregnancies (except those conceived through in vitro fertilization) have a margin of error of at least 5 days.²⁰ Third, our data do not provide information on reasons for withholding or withdrawing care, and actively treated infants may be different in unmeasured ways from their counterparts despite having similar probabilities of survival based on our model.³⁹ Decisions about whether to provide life support at the margins of viability involve more than statistical calculations and must take into account clinical judgment and the values, beliefs, and concerns of each family.⁴⁰ Fourth, prognosis is dynamic and can change with new events after birth⁴¹; the outcomes here reflect prognostic estimates for infants at the time of birth. Fifth, we did not have data on the causes of hospital variation in outcomes. Further research is needed to identify explanations for the observed variation. Sixth, although we evaluated the performance of the updated model among infants born several years after those whose data were used for model development, we were unable to provide estimates regarding how often the model should be periodically recalibrated to account for changing outcomes over time.

Our final model explicitly incorporated the hospital of birth in outcome estimates, which was possible because VON included data for most eligible extremely preterm infants born in the United States during the period. Hospital-specific survival estimates for each hospital participating (eAppendix in Supplement) in VON are available to that hospital through the VON website.⁴² The NICHD has published an updated model providing the mean and range of hospital outcomes on their website available to the public.⁴³

Conclusions

The updated NICHD NRN extremely preterm birth outcome model continues to demonstrate the importance of considering more than gestational age when evaluating prognosis at the time of birth. The hospital of birth contributed substantially to outcome prediction. As outcomes and practices continue to change, prognostic models used for extremely preterm birth may require accounting for local outcomes and periodic updating to remain relevant.

Supplement.

eFigure 1. Updated Probabilities of Survival for Infants in the NRN and VON 2006-2012 Cohorts, by Gestational Age at Birth

eFigure 2. Model Calibration Plots

eFigure 3. Decision Curve Analyses

eFigure 4. Differences in Observed and Expected Outcomes in 2006-2012 by Hospital, Without Accounting for Hospital Differences

eFigure 5. Differences in Observed and Expected Outcomes in 2013-2016 by Hospital, Using Hospital-Specific Intercepts Derived 2006-2012

eTable 1. Resources Utilized per Surviving Infant by Probability of Survival

eTable 2. Neurodevelopmental Outcomes Among Infants at Follow-up by Probability of Survival

eTable 3. Observed and Predicted Rates of Survival for Actively Treated and All Infants

eAppendix. List of Participating Hospitals in the Vermont Oxford Network

Click here for additional data file.^{(1,018.2KB, pdf)}

References

1.Rysavy MA, Tyson JE. The problem and promise of prognosis research. JAMA Pediatr. 2016;170(5):411-412. doi: 10.1001/jamapediatrics.2015.4871 [DOI] [PubMed] [Google Scholar]
2.Rysavy MA. Prognosis as an intervention. Clin Perinatol. 2018;45(2):231-240. doi: 10.1016/j.clp.2018.01.009 [DOI] [PubMed] [Google Scholar]
3.NICHD Neonatal Research Network (NRN) Extremely preterm birth outcome data. https://www.nichd.nih.gov/about/org/der/branches/ppb/programs/epbo/Pages/epbo_case.aspx. Accessed July 1, 2019.
4.Tyson JE, Parikh NA, Langer J, Green C, Higgins RD; National Institute of Child Health and Human Development Neonatal Research Network . Intensive care for extreme prematurity: moving beyond gestational age. N Engl J Med. 2008;358(16):1672-1681. doi: 10.1056/NEJMoa073059 [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Myers P, Laventhal N, Andrews B, Lagatta J, Meadow W. Population-based outcomes data for counseling at the margin of gestational viability. J Pediatr. 2017;181:208-212.e4. doi: 10.1016/j.jpeds.2016.10.021 [DOI] [PubMed] [Google Scholar]
6.American College of Obstetricians and Gynecologists, Society for Maternal-Fetal Medicine. Obstetric Care Consensus No. 6: periviable birth. Obstet Gynecol. 2017;130(4):e187-e199. . doi: 10.1097/AOG.0000000000002352 [DOI] [PubMed] [Google Scholar]
7.Cummings J; Committee on Fetus and Newborn . Antenatal counseling regarding resuscitation and intensive care before 25 weeks of gestation. Pediatrics. 2015;136(3):588-595. doi: 10.1542/peds.2015-2336 [DOI] [PubMed] [Google Scholar]
8.Horbar JD, Carpenter JH, Badger GJ, et al. . Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009. Pediatrics. 2012;129(6):1019-1026. doi: 10.1542/peds.2011-3028 [DOI] [PubMed] [Google Scholar]
9.Patel RM, Kandefer S, Walsh MC, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Causes and timing of death in extremely premature infants from 2000 through 2011. N Engl J Med. 2015;372(4):331-340. doi: 10.1056/NEJMoa1403489 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Horbar JD, Onstad L. Survival for inborn infants weighing 501-1500 grams at birth: Variation among centers. Pediatr Res. 1990;27(4):245A.2108426 [Google Scholar]
11.Stoll BJ, Hansen NI, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network. Pediatrics. 2010;126(3):443-456. doi: 10.1542/peds.2009-2959 [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Alleman BW, Bell EF, Li L, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Individual and center-level factors affecting mortality among extremely low birth weight infants. Pediatrics. 2013;132(1):e175-e184. doi: 10.1542/peds.2012-3707 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Horbar JD, Edwards EM, Greenberg LT, et al. . Variation in performance of neonatal intensive care units in the United States. JAMA Pediatr. 2017;171(3):e164396. doi: 10.1001/jamapediatrics.2016.4396 [DOI] [PubMed] [Google Scholar]
14.Kramer MS, McLean FH, Boyd ME, Usher RH. The validity of gestational age estimation by menstrual dating in term, preterm, and postterm gestations. JAMA. 1988;260(22):3306-3308. doi: 10.1001/jama.1988.03410220090034 [DOI] [PubMed] [Google Scholar]
15.Kramer MS, Platt RW, Wen SW, et al. ; Fetal/Infant Health Study Group of the Canadian Perinatal Surveillance System . A new and improved population-based Canadian reference for birth weight for gestational age. Pediatrics. 2001;108(2):E35. doi: 10.1542/peds.108.2.e35 [DOI] [PubMed] [Google Scholar]
16.Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594. doi: 10.1136/bmj.g7594 [DOI] [PubMed] [Google Scholar]
17.Rysavy MA, Marlow N, Doyle LW, et al. . Reporting outcomes of extremely preterm births. Pediatrics. 2016;138(3):e20160689. doi: 10.1542/peds.2016-0689 [DOI] [PubMed] [Google Scholar]
18.Rysavy MA, Li L, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Between-hospital variation in treatment and outcomes in extremely preterm infants. N Engl J Med. 2015;372(19):1801-1811. doi: 10.1056/NEJMoa1410689 [DOI] [PMC free article] [PubMed] [Google Scholar]
19.United States Department of Health and Human Services (US DHHS) CDC WONDER online database: natality public-use data 2007-2017. https://wonder.cdc.gov/natality-current.html. Accessed Feb 1, 2019.
20.Spong CY. Defining “term” pregnancy: recommendations from the Defining “Term” Pregnancy Workgroup. JAMA. 2013;309(23):2445-2446. doi: 10.1001/jama.2013.6235 [DOI] [PubMed] [Google Scholar]
21.Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361-387. doi: [DOI] [PubMed] [Google Scholar]
22.Riley RD, Ensor J, Snell KIE, et al. . External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016;353:i3140. doi: 10.1136/bmj.i3140 [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Guillén Ú, Weiss EM, Munson D, et al. . Guidelines for the management of extremely premature deliveries: a systematic review. Pediatrics. 2015;136(2):343-350. doi: 10.1542/peds.2015-0542 [DOI] [PubMed] [Google Scholar]
24.Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6. doi: 10.1136/bmj.i6 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565-574. doi: 10.1177/0272989X06295361 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Puopolo KM, Draper D, Wi S, et al. . Estimating the probability of neonatal early-onset infection on the basis of maternal risk factors. Pediatrics. 2011;128(5):e1155-e1163. doi: 10.1542/peds.2010-3464 [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Marlow N. Keeping up with outcomes for infants born at extremely low gestational ages. JAMA Pediatr. 2015;169(3):207-208. doi: 10.1001/jamapediatrics.2014.3362 [DOI] [PubMed] [Google Scholar]
28.Bouwmeester W, Twisk JW, Kappen TH, van Klei WA, Moons KG, Vergouwe Y. Prediction models for clustered data: comparison of a random intercept and standard regression model. BMC Med Res Methodol. 2013;13:19. doi: 10.1186/1471-2288-13-19 [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Wynants L, Vergouwe Y, Van Huffel S, Timmerman D, Van Calster B. Does ignoring clustering in multicenter data influence the performance of prediction models? a simulation study. Stat Methods Med Res. 2018;27(6):1723-1736. doi: 10.1177/0962280216668555 [DOI] [PubMed] [Google Scholar]
30.Wynants L, Kent DM, Timmerman D, Lundquist CM, Van Calster B. Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting. Diagn Progn Res. 2019;3:6. doi: 10.1186/s41512-019-0046-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
31.Rogowski JA, Horbar JD, Staiger DO, Kenny M, Carpenter J, Geppert J. Indirect vs direct hospital quality indicators for very low-birth-weight infants. JAMA. 2004;291(2):202-209. doi: 10.1001/jama.291.2.202 [DOI] [PubMed] [Google Scholar]
32.Boland RA, Davis PG, Dawson JA, Doyle LW; Victorian Infant Collaborative Study Group . Predicting death or major neurodevelopmental disability in extremely preterm infants born in Australia. Arch Dis Child Fetal Neonatal Ed. 2013;98(3):F201-F204. doi: 10.1136/archdischild-2012-301628 [DOI] [PubMed] [Google Scholar]
33.Marrs CC, Pedroza C, Mendez-Figueroa H, Chauhan SP, Tyson JE. Infant outcomes after periviable birth: external validation of the neonatal research network estimator with the BEAM trial. Am J Perinatol. 2016;33(6):569-576. [DOI] [PubMed] [Google Scholar]
34.Cross G, Becker M, Congdon P. Prognosis for babies born with fused eyelids. Arch Dis Child. 1985;60(5):479-480. doi: 10.1136/adc.60.5.479 [DOI] [PMC free article] [PubMed] [Google Scholar]
35.Stefano JL, Morales M. Fused eyelids in the extremely premature infant: multivariate analysis of survival and outcome. Am J Perinatol. 1992;9(2):84-86. doi: 10.1055/s-2007-994677 [DOI] [PubMed] [Google Scholar]
36.Ngo TTM, Moufarrej MN, Rasmussen MH, et al. . Noninvasive blood tests for fetal development predict gestational age and preterm delivery. Science. 2018;360(6393):1133-1136. doi: 10.1126/science.aar3819 [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Stoll BJ, Hansen NI, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012. JAMA. 2015;314(10):1039-1051. doi: 10.1001/jama.2015.10244 [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Skupski DW, McCullough LB, Levene M, Chervenak FA. Improving obstetric estimation of outcomes of extremely premature neonates: an evolving challenge. J Perinat Med. 2010;38(1):19-22. doi: 10.1515/jpm.2010.013 [DOI] [PubMed] [Google Scholar]
39.Atwell K, Callander E, Lindsay D, Marshall PB, Morris SA. Selection bias and outcomes for preterm neonates. Pediatrics. 2018;142(1):e20180470. doi: 10.1542/peds.2018-0470 [DOI] [PubMed] [Google Scholar]
40.Lantos JD. Ethical problems in decision making in the neonatal ICU. N Engl J Med. 2018;379(19):1851-1860. doi: 10.1056/NEJMra1801063 [DOI] [PubMed] [Google Scholar]
41.Ambalavanan N, Carlo WA, Tyson JE, et al. ; Generic Database; Subcommittees of the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Outcome trajectories in extremely preterm infants. Pediatrics. 2012;130(1):e115-e125. doi: 10.1542/peds.2011-3693 [DOI] [PMC free article] [PubMed] [Google Scholar]
42.Vermont Oxford Network Home page. https://nightingale.vtoxford.org. Accessed January 28, 2020.
43.Eunice Kennedy Shriver National Institute of Child Health and Human Development Home page. https://www.nichd.nih.gov/research/supported/EPBO. Accessed January 28, 2020.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement.

eFigure 1. Updated Probabilities of Survival for Infants in the NRN and VON 2006-2012 Cohorts, by Gestational Age at Birth

eFigure 2. Model Calibration Plots

eFigure 3. Decision Curve Analyses

eFigure 4. Differences in Observed and Expected Outcomes in 2006-2012 by Hospital, Without Accounting for Hospital Differences

eFigure 5. Differences in Observed and Expected Outcomes in 2013-2016 by Hospital, Using Hospital-Specific Intercepts Derived 2006-2012

eTable 1. Resources Utilized per Surviving Infant by Probability of Survival

eTable 2. Neurodevelopmental Outcomes Among Infants at Follow-up by Probability of Survival

eTable 3. Observed and Predicted Rates of Survival for Actively Treated and All Infants

eAppendix. List of Participating Hospitals in the Vermont Oxford Network

Click here for additional data file.^{(1,018.2KB, pdf)}

[poi190111r1] 1.Rysavy MA, Tyson JE. The problem and promise of prognosis research. JAMA Pediatr. 2016;170(5):411-412. doi: 10.1001/jamapediatrics.2015.4871 [DOI] [PubMed] [Google Scholar]

[poi190111r2] 2.Rysavy MA. Prognosis as an intervention. Clin Perinatol. 2018;45(2):231-240. doi: 10.1016/j.clp.2018.01.009 [DOI] [PubMed] [Google Scholar]

[poi190111r3] 3.NICHD Neonatal Research Network (NRN) Extremely preterm birth outcome data. https://www.nichd.nih.gov/about/org/der/branches/ppb/programs/epbo/Pages/epbo_case.aspx. Accessed July 1, 2019.

[poi190111r4] 4.Tyson JE, Parikh NA, Langer J, Green C, Higgins RD; National Institute of Child Health and Human Development Neonatal Research Network . Intensive care for extreme prematurity: moving beyond gestational age. N Engl J Med. 2008;358(16):1672-1681. doi: 10.1056/NEJMoa073059 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r5] 5.Myers P, Laventhal N, Andrews B, Lagatta J, Meadow W. Population-based outcomes data for counseling at the margin of gestational viability. J Pediatr. 2017;181:208-212.e4. doi: 10.1016/j.jpeds.2016.10.021 [DOI] [PubMed] [Google Scholar]

[poi190111r6] 6.American College of Obstetricians and Gynecologists, Society for Maternal-Fetal Medicine. Obstetric Care Consensus No. 6: periviable birth. Obstet Gynecol. 2017;130(4):e187-e199. . doi: 10.1097/AOG.0000000000002352 [DOI] [PubMed] [Google Scholar]

[poi190111r7] 7.Cummings J; Committee on Fetus and Newborn . Antenatal counseling regarding resuscitation and intensive care before 25 weeks of gestation. Pediatrics. 2015;136(3):588-595. doi: 10.1542/peds.2015-2336 [DOI] [PubMed] [Google Scholar]

[poi190111r8] 8.Horbar JD, Carpenter JH, Badger GJ, et al. . Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009. Pediatrics. 2012;129(6):1019-1026. doi: 10.1542/peds.2011-3028 [DOI] [PubMed] [Google Scholar]

[poi190111r9] 9.Patel RM, Kandefer S, Walsh MC, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Causes and timing of death in extremely premature infants from 2000 through 2011. N Engl J Med. 2015;372(4):331-340. doi: 10.1056/NEJMoa1403489 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r10] 10.Horbar JD, Onstad L. Survival for inborn infants weighing 501-1500 grams at birth: Variation among centers. Pediatr Res. 1990;27(4):245A.2108426 [Google Scholar]

[poi190111r11] 11.Stoll BJ, Hansen NI, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network. Pediatrics. 2010;126(3):443-456. doi: 10.1542/peds.2009-2959 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r12] 12.Alleman BW, Bell EF, Li L, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Individual and center-level factors affecting mortality among extremely low birth weight infants. Pediatrics. 2013;132(1):e175-e184. doi: 10.1542/peds.2012-3707 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r13] 13.Horbar JD, Edwards EM, Greenberg LT, et al. . Variation in performance of neonatal intensive care units in the United States. JAMA Pediatr. 2017;171(3):e164396. doi: 10.1001/jamapediatrics.2016.4396 [DOI] [PubMed] [Google Scholar]

[poi190111r14] 14.Kramer MS, McLean FH, Boyd ME, Usher RH. The validity of gestational age estimation by menstrual dating in term, preterm, and postterm gestations. JAMA. 1988;260(22):3306-3308. doi: 10.1001/jama.1988.03410220090034 [DOI] [PubMed] [Google Scholar]

[poi190111r15] 15.Kramer MS, Platt RW, Wen SW, et al. ; Fetal/Infant Health Study Group of the Canadian Perinatal Surveillance System . A new and improved population-based Canadian reference for birth weight for gestational age. Pediatrics. 2001;108(2):E35. doi: 10.1542/peds.108.2.e35 [DOI] [PubMed] [Google Scholar]

[poi190111r16] 16.Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594. doi: 10.1136/bmj.g7594 [DOI] [PubMed] [Google Scholar]

[poi190111r17] 17.Rysavy MA, Marlow N, Doyle LW, et al. . Reporting outcomes of extremely preterm births. Pediatrics. 2016;138(3):e20160689. doi: 10.1542/peds.2016-0689 [DOI] [PubMed] [Google Scholar]

[poi190111r18] 18.Rysavy MA, Li L, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Between-hospital variation in treatment and outcomes in extremely preterm infants. N Engl J Med. 2015;372(19):1801-1811. doi: 10.1056/NEJMoa1410689 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r19] 19.United States Department of Health and Human Services (US DHHS) CDC WONDER online database: natality public-use data 2007-2017. https://wonder.cdc.gov/natality-current.html. Accessed Feb 1, 2019.

[poi190111r20] 20.Spong CY. Defining “term” pregnancy: recommendations from the Defining “Term” Pregnancy Workgroup. JAMA. 2013;309(23):2445-2446. doi: 10.1001/jama.2013.6235 [DOI] [PubMed] [Google Scholar]

[poi190111r21] 21.Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361-387. doi: [DOI] [PubMed] [Google Scholar]

[poi190111r22] 22.Riley RD, Ensor J, Snell KIE, et al. . External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016;353:i3140. doi: 10.1136/bmj.i3140 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r23] 23.Guillén Ú, Weiss EM, Munson D, et al. . Guidelines for the management of extremely premature deliveries: a systematic review. Pediatrics. 2015;136(2):343-350. doi: 10.1542/peds.2015-0542 [DOI] [PubMed] [Google Scholar]

[poi190111r24] 24.Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6. doi: 10.1136/bmj.i6 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r25] 25.Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565-574. doi: 10.1177/0272989X06295361 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r26] 26.Puopolo KM, Draper D, Wi S, et al. . Estimating the probability of neonatal early-onset infection on the basis of maternal risk factors. Pediatrics. 2011;128(5):e1155-e1163. doi: 10.1542/peds.2010-3464 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r27] 27.Marlow N. Keeping up with outcomes for infants born at extremely low gestational ages. JAMA Pediatr. 2015;169(3):207-208. doi: 10.1001/jamapediatrics.2014.3362 [DOI] [PubMed] [Google Scholar]

[poi190111r28] 28.Bouwmeester W, Twisk JW, Kappen TH, van Klei WA, Moons KG, Vergouwe Y. Prediction models for clustered data: comparison of a random intercept and standard regression model. BMC Med Res Methodol. 2013;13:19. doi: 10.1186/1471-2288-13-19 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r29] 29.Wynants L, Vergouwe Y, Van Huffel S, Timmerman D, Van Calster B. Does ignoring clustering in multicenter data influence the performance of prediction models? a simulation study. Stat Methods Med Res. 2018;27(6):1723-1736. doi: 10.1177/0962280216668555 [DOI] [PubMed] [Google Scholar]

[poi190111r30] 30.Wynants L, Kent DM, Timmerman D, Lundquist CM, Van Calster B. Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting. Diagn Progn Res. 2019;3:6. doi: 10.1186/s41512-019-0046-9 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r31] 31.Rogowski JA, Horbar JD, Staiger DO, Kenny M, Carpenter J, Geppert J. Indirect vs direct hospital quality indicators for very low-birth-weight infants. JAMA. 2004;291(2):202-209. doi: 10.1001/jama.291.2.202 [DOI] [PubMed] [Google Scholar]

[poi190111r32] 32.Boland RA, Davis PG, Dawson JA, Doyle LW; Victorian Infant Collaborative Study Group . Predicting death or major neurodevelopmental disability in extremely preterm infants born in Australia. Arch Dis Child Fetal Neonatal Ed. 2013;98(3):F201-F204. doi: 10.1136/archdischild-2012-301628 [DOI] [PubMed] [Google Scholar]

[poi190111r33] 33.Marrs CC, Pedroza C, Mendez-Figueroa H, Chauhan SP, Tyson JE. Infant outcomes after periviable birth: external validation of the neonatal research network estimator with the BEAM trial. Am J Perinatol. 2016;33(6):569-576. [DOI] [PubMed] [Google Scholar]

[poi190111r34] 34.Cross G, Becker M, Congdon P. Prognosis for babies born with fused eyelids. Arch Dis Child. 1985;60(5):479-480. doi: 10.1136/adc.60.5.479 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r35] 35.Stefano JL, Morales M. Fused eyelids in the extremely premature infant: multivariate analysis of survival and outcome. Am J Perinatol. 1992;9(2):84-86. doi: 10.1055/s-2007-994677 [DOI] [PubMed] [Google Scholar]

[poi190111r36] 36.Ngo TTM, Moufarrej MN, Rasmussen MH, et al. . Noninvasive blood tests for fetal development predict gestational age and preterm delivery. Science. 2018;360(6393):1133-1136. doi: 10.1126/science.aar3819 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r37] 37.Stoll BJ, Hansen NI, Bell EF, et al. ; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012. JAMA. 2015;314(10):1039-1051. doi: 10.1001/jama.2015.10244 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r38] 38.Skupski DW, McCullough LB, Levene M, Chervenak FA. Improving obstetric estimation of outcomes of extremely premature neonates: an evolving challenge. J Perinat Med. 2010;38(1):19-22. doi: 10.1515/jpm.2010.013 [DOI] [PubMed] [Google Scholar]

[poi190111r39] 39.Atwell K, Callander E, Lindsay D, Marshall PB, Morris SA. Selection bias and outcomes for preterm neonates. Pediatrics. 2018;142(1):e20180470. doi: 10.1542/peds.2018-0470 [DOI] [PubMed] [Google Scholar]

[poi190111r40] 40.Lantos JD. Ethical problems in decision making in the neonatal ICU. N Engl J Med. 2018;379(19):1851-1860. doi: 10.1056/NEJMra1801063 [DOI] [PubMed] [Google Scholar]

[poi190111r41] 41.Ambalavanan N, Carlo WA, Tyson JE, et al. ; Generic Database; Subcommittees of the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network . Outcome trajectories in extremely preterm infants. Pediatrics. 2012;130(1):e115-e125. doi: 10.1542/peds.2011-3693 [DOI] [PMC free article] [PubMed] [Google Scholar]

[poi190111r42] 42.Vermont Oxford Network Home page. https://nightingale.vtoxford.org. Accessed January 28, 2020.

[poi190111r43] 43.Eunice Kennedy Shriver National Institute of Child Health and Human Development Home page. https://www.nichd.nih.gov/research/supported/EPBO. Accessed January 28, 2020.

PERMALINK

Assessment of an Updated Neonatal Research Network Extremely Preterm Birth Outcome Model in the Vermont Oxford Network

Matthew A Rysavy, MD, PhD

Jeffrey D Horbar, MD

Edward F Bell, MD

Lei Li, PhD

Lucy T Greenberg, MS

Jon E Tyson, MD, MPH

Ravi M Patel, MD, MS

Waldemar A Carlo, MD

Noelle E Younge, MD, MHS

Charles E Green, PhD

Erika M Edwards, PhD

Susan R Hintz, MD, MS

Michele C Walsh, MD, MS

Jeffrey S Buzas, PhD

Abhik Das, PhD

Rosemary D Higgins, MD

Key Points

Question

Findings

Meaning

Abstract

Importance

Objective

Design, Setting, and Participants

Exposures

Main Outcomes and Measures

Results

Conclusions and Relevance

Introduction

Methods

Derivation Cohort

Validation Cohorts

Predictive Variables

Outcomes

Statistical Analysis

Results

Table 1. Characteristics of Actively Treated Infants in Each Cohorta.

Model Derivation

Figure 1. Odds Ratios for Survival in the Original and Updated Models.

Model Validation

Table 2. Comparison of the Updated and Original 5-Factor Models in the VON 2006-2012 Cohort.

Figure 2. Observed and Model-Predicted Survival Rates in the Vermont Oxford Network (VON) 2006-2012 Cohort.

Hospital of Birth

Table 3. Relative Contribution of Variables to the Multivariable Modela.

Sensitivity Analyses

Discussion

Limitations

Conclusions

References

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases

Table 1. Characteristics of Actively Treated Infants in Each Cohort^a.

Table 3. Relative Contribution of Variables to the Multivariable Model^a.