Development of Modified Screening Criteria for Retinopathy of Prematurity

Key Points

Question

Can a highly sensitive, postnatal weight gain–based retinopathy of prematurity predictive model be developed using data from a diverse cohort of at-risk infants?

Findings

In this cohort study of 7483 premature infants, a model consisting of 6 criteria correctly predicted 100% of infants with type 1 retinopathy of prematurity while reducing the number of infants who required examinations by 30.3% when only infants who met the criteria received examinations.

Meaning

If validated, these modified screening criteria could be used to reduce the number of infants who require examinations while consistently identifying treatment-requiring retinopathy of prematurity.

Abstract

Importance

Current retinopathy of prematurity (ROP) guidelines, which are based on studies of high-risk infants and expert opinion, have low specificity for disease requiring treatment. Postnatal weight gain–based models improve specificity but have been limited by complexity and small development cohorts, which results in model overfitting and resultant decreased sensitivity in validation studies.

Objective

To develop a birth weight (BW), gestational age (GA), and weight gain (WG) prediction model using data from a broad-risk cohort of premature infants.

Design, Setting, and Participants

The Postnatal Growth and ROP Study was a retrospective multicenter cohort study conducted in 29 hospitals in the United States and Canada from 2006 to 2012 that included 7483 premature infants at risk for ROP with a known ROP outcome. A hybrid modeling approach was used that combined BW/GA criteria, weight comparison with expected growth from infants without ROP, multiple growth-interval assessments, consideration of nonphysiological WG, and user-friendly screening criteria. Numerous BW/GA levels, postnatal age periods, time intervals, and WG percentile thresholds were evaluated to identify the most robust parameters.

Main Outcome and Measures

Sensitivity for Early Treatment of ROP Study type 1 ROP and potential reduction in infants who require examinations.

Results

Of 7483 infants, the median (SD) BW was 1099 (359) g, the median GA was 28 weeks (range, 22-35), 3575 (47.8%) were female, 3615 (48.4%) were white, 2310 (30.9%) were black, 233 (3.1%) were Asian, 93 (1.2%) were Pacific Islander, and 40 (0.5%) were American Indian/Alaskan Native. Infants who met any of 6 criteria would undergo examinations: (1) a GA of younger than 28 weeks; (2) a BW of less than 1051 g; a WG of less than 120 g, 180 g, or 170 g during ages 10 to 19, 20 to 29, or 30 to 39 days, respectively; or hydrocephalus. These criteria predicted 459 of 459 (100%) type 1 (sensitivity, 100%; 95% CI, 99.2%-100%), 524 of 524 (100%) treated, and 466 of 472 (98.7%) type 2 cases while reducing the number of infants who required examinations by 2269 (30.3%).

Conclusions and Relevance

This cohort study, broadly representative of infants who are undergoing ROP examinations, provides evidence-based screening criteria. With validation, the Postnatal Growth and ROP Study criteria could be incorporated into ROP screening guidelines to reduce the number of infants who require examinations in North America.

This cohort study aims to develop evidence-based retinopathy of prematurity screening criteria using data on birth weight, gestational age, and weight gain from a cohort of at-risk infants in North America.

Introduction

Retinopathy of prematurity (ROP) is a blinding disease of the developing retinal vasculature. Clinical management includes serial retinal examinations of at-risk infants and treatment with laser retinal photocoagulation or intravitreal injection of an antivascular endothelial growth factor agent to reduce the risk of progression to retinal detachment.^1,2 Current ROP screening criteria are based on birth weight (BW) and gestational age at birth (GA) (eg, in the United States, BW < 1501 g or GA ≤ 30 weeks).^3,4 Approximately 70 000 infants a year in the United States alone receive examinations.⁵ The current screening criteria have low specificity for predicting which infants are at risk for severe ROP; only 5% to 10% of infants who are examined require treatment.^{1,6,7,8,9,10,11} In addition, while their sensitivity for predicting severe ROP is very high, it is not 100%, as larger-BW and older-GA infants sometimes require treatment, and the guidelines include a third, poorly defined screening criterion for larger-BW, older-GA infants with a poor postnatal course in the judgement of the neonatologist.^3,4 Therefore, there are opportunities to improve both the specificity and sensitivity of these criteria if risk factors for ROP other than BW and GA could be applied in a more systematic manner.

Advances in the pathogenesis of ROP have led to the development of predictive models that include slow postnatal weight gain as a predictor of ROP.^{12,13,14,15,16,17,18,19,20} Slow weight gain is a presumed surrogate for low serum insulin–like growth factor 1 (IGF-1) levels, which result in poor vascular endothelial growth factor–mediated retinal vascular growth and lead to the development of ROP.^21,22,23 Predictive models that incorporate postnatal weight gain have included WINROP,^12,13,14 Premature Infants in Need of Transfusion ROP (PINT ROP),¹⁵ ROPScore,¹⁶ Children’s Hospital of Philadelphia ROP (CHOP ROP),^17,24 and Colorado ROP (CO-ROP).^18,19 While varying statistical approaches were used in developing each model, sensitivities of 100% for predicting severe ROP with large potential decreases in the number of infants who required examinations were initially reported. However, the development of a prediction model should be performed using as large a cohort as possible to avoid overfitting of the model to the data or the model will not perform well when validated in new participants.^25,26,27 The weight gain models reported to date were developed using cohorts that were small, resulting in model overfitting and decreased sensitivity for predicting severe ROP in subsequent validation studies.^13,19,24 Without high confidence that infants who require treatment will receive examinations, none of the models could be proposed as a replacement for the current screening criteria. To avoid overfitting, a much larger development cohort that contains hundreds of infants with severe ROP is required.^28,29 Complex calculations further limited the clinical implementation of most of the models.²⁹ To gain widespread acceptance, a model should be transparent and easy to use.^26,30

We sought to develop a new BW, GA, and postnatal weight gain ROP model using data from a diverse cohort of at-risk infants in North America. We planned a cohort large enough to minimize overfitting and provide precise estimates of sensitivity and aimed to develop a model that was simple enough to be applied clinically.

Methods

We conducted a multicenter study called the Postnatal Growth and ROP (G-ROP) Study.²⁸ The primary aim of the G-ROP Study was to develop a growth-based ROP predictive model, and this article presents those primary results. The study design was a retrospective cohort study. We collected data retrospectively on infants who underwent ROP examinations at 29 hospitals in the United States and Canada. Institutional review board approval was obtained and a waiver of consent was granted at all of the hospitals. Eligible infants were those who were born between January 1, 2006, and December 31, 2011, who underwent ROP examinations and had a known ROP outcome. Specific BW and GA limits were not used for the G-ROP Study to make the cohort fully representative of all infants who were undergoing ROP examinations. However, typical criteria used during the study period included a BW of less than 1501 g, a GA of 30 weeks or younger, or an unstable clinical course as determined by the neonatologist.²⁸ Infants were considered to have a known ROP outcome if (1) either eye had Early Treatment of ROP (ETROP) Study type 1 ROP, type 2 ROP, or ROP treatment or (2) both eyes had retinal vasculature maturity, immature vasculature extending into zone III without prior disease in zone I or II, or regression of ROP not reaching criteria for type 1 or 2 ROP.²⁸ Certified data abstractors in the G-ROP Study collected detailed ophthalmologic and medical data, including BW, GA, and daily postnatal weight measurements, from medical records. The data were entered into a web-based database. Data quality was ensured through data entry validation rules, data audits, and discrepancy check algorithms, the details of which have been published.²⁸ All flagged values were investigated and resolved.

Statistical Analysis

We applied a hybrid modeling approach that combined elements of prior models, including BW and GA thresholds (current screening guidelines³), comparison with expected growth (WINROP³¹), discrete growth periods (ROPScore¹⁶ and CO-ROP¹⁸), multiple growth intervals (WINROP³¹ and CHOP ROP¹⁷), nonphysiological weight gain confounders, and user-friendly criteria (current guidelines³ and CO-ROP¹⁸). A brief review of the structures of these models is helpful to understand the methods that we used.

The current BW and GA screening levels are a simple prediction model, which is an equation or criteria that predict an outcome, such as ROP.^26,29 WINROP was the first model to incorporate postnatal weight gain into the prediction of ROP.^31,32 The algorithm involves weekly comparisons of observed weights with a growth curve of expected weights that is derived from a cohort of infants who developed mild or no ROP. The weekly differences between expected and observed weights are added until the total surpasses a threshold alarm level. The CHOP ROP model is simpler and consists of a single logistic regression–based equation with terms for BW, GA, and weight gain rate based on the preceding week’s weight measurements.¹⁷ The risk of severe ROP is recalculated using the CHOP ROP model on a weekly basis. If the risk is above an alarm level, examinations are indicated. ROPScore also consists of a regression-based equation but considers weight gain only once, at age 6 weeks.¹⁶ The CO-ROP model is a further simplification that considers weight gain once at age 4 weeks but takes the form of criteria rather than an equation.¹⁸

For this study, model development began by lowering the BW and GA levels used for current ROP guidelines. Numerous combinations of thresholds were evaluated: BW values from 500 g to 1500 g in 50-g intervals, and GA values from 25 weeks to 30 weeks in 1-week intervals. Lowering these thresholds results in examining fewer infants, but additional factors, such as slow weight gain, must be added to correctly predict the cases of severe ROP that the lowered thresholds will no longer capture. Weight gain was incorporated into the model through a comparison of observed weights with expected weights, as with WINROP. Expected weight gain was defined using the postnatal weight measurements of the 4259 infants (57.3%) in the G-ROP Study who did not develop ROP. Slow weight gain was defined as an observed weight gain of less than a specified percentile of the distribution of expected weight gain; all levels were evaluated from the 5th percentile through the 50th percentile, in 5-percentile increments. The period during which the comparisons of observed weight with expected weight were made was determined by identifying a period during postnatal life when the rates of daily weight gain for infants with and without ETROP types 1 or 2 ROP were clearly differentiable, or approximately age 10 through 40 days (Figure 1). This period was defined in terms of chronological age (age since birth) rather than developmental age (postmenstrual age), even though the course of ROP is tied closely to developmental age, because weight gain calculations based on chronological age are easier for clinicians to perform and older-GA infants may not have weight data available at early postmenstrual ages. Varying numbers of comparison growth period intervals (1-5) and interval lengths (eg, 7 days, 10 days, 14 days) were considered within this overall period.

Figure 1. Mean Daily Weight Gain by Chronological Age of 7483 Infants in the Postnatal Growth and Retinopathy of Prematurity Study.

All of the previously mentioned factors (BW levels, GA levels, percentiles of expected weight gain below which observed weight gain would be considered as “slow,” the number of time intervals during which slow growth is identified, and interval lengths) were considered simultaneously. Thousands of combinations were evaluated to identify criteria that correctly predicted all infants who developed type 1 ROP and maximally reduced the number of infants who would receive examinations if the model were to be used clinically to decide which infants received examinations. Potential sources of nonphysiological weight gain (weight gain despite low IGF levels) were considered as confounders that might cause “false-negative” signals from the model, including hydrocephalus, which was previously reported as one such factor.¹³ Finally, ease of use was prioritized in constructing the model because successful clinical implementation rests partly on simplicity and transparency.^26,30 The familiar structure of screening criteria was preferred to an equation or algorithm. Calculations were kept as simple as possible by using absolute weight gain rather than weight gain rate, choosing the smallest number of growth intervals that still discriminated well between a low and high risk of ROP, and using round numbers of days for interval size.

The primary study outcomes were the sensitivity for predicting ETROP type 1 ROP (the proportion of infants who developed type 1 disease in 1 or both eyes for whom examinations would be indicated by the model) and the reduction in infants who receive examinations, which is a more intuitive measure of model specificity. For these assessments, the model was used to make “all or none” ROP screening decisions (ie, infants who met the screening criteria established by the model would receive examinations, and the remaining infants would not receive examinations). Secondary outcomes included sensitivities for type 2 ROP, type 1 or 2 ROP, and treated ROP. The 95% confidence intervals for sensitivity were calculated using the Wilson method.³³ Analyses were performed using SAS, version 9.3 (SAS Institute).

Results

The G-ROP Study cohort included 7483 infants (Table). Retinopathy of prematurity developed in 3324 infants (43%), of whom 459 (6.1%) developed type 1 ROP, 472 (6.3%) developed type 2 ROP, and 524 (7%) were treated. The median BW was 1080 g (range, 310-3000 g), and the median GA was 28 weeks (range, 22-35 weeks). Of the patients, 3575 (47.8%) were female, 3615 (48.4%) were white, 2310 (30.9%) were black, 564 (7.5%) were Latino, 233 (3.1%) were Asian, 93 (1.2%) were Native Hawaiian or Pacific Islander, and 40 (0.5%) were American Indian or Alaskan Native infants; 5612 infants (73.7%) were born at a study hospital.

Table. Birth Weight and Gestational Age at Birth of 7483 Infants in the Postnatal Growth and Retinopathy of Prematurity Study.

Characteristic	All Patients (N = 7483)	Type 1 ROP (n = 459)	Type 2 ROP (n = 472)	ROP Other (n = 2293)	No ROP (n = 4259)
Birth weight, g
Mean (SD)	1099 (359)	714 (205)	748 (191)	927 (273)	1274 (327)
Median (1st quartile, 3rd quartile) [range]	1070 (810, 1358) [310-3000]	668 (573, 817) [310-1692]	720 (612, 844) [372-1590]	880 (730, 1076) [364-2880]	1265 (1050, 1470) [400-3000]
Gestational age, wk
Mean (SD)	28 (3)	25 (1.6)	25 (1.6)	27 (2.1)	29 (2.1)
Median (1st quartile, 3rd quartile) [range]	28 (26,30) [22-35]	25 (24, 26) [22-31]	25 (24, 26) [22-32]	27 (25, 28) [22-35]	30 (28, 31) [23-35]

Abbreviation: ROP, retinopathy of prematurity.

The final model consisted of 6 ROP screening criteria: a BW of less than 1051 g; a GA of less than 28 weeks, 0 days; a weight gain of less than 120 g during the second 10 days following birth (age 10-19 days), 180 g during the third 10 days following birth (age 20-29 days), or 170 g during the fourth 10 days following birth (age 30-39 days); or hydrocephalus diagnosed on brain imaging study (ultrasonography, computed tomography, or magnetic resonance imaging). These criteria would be used in a fashion similar to the currently used screening criteria (Figure 2). Infants meeting 1 or more of these criteria would be considered at high risk for severe ROP and would undergo examinations; infants who did not meet any of the 6 screening criteria would not undergo examinations. Applied in this fashion, the model correctly predicted 459 of 459 infants with type 1 ROP (sensitivity, 100%; 95% CI, 99.2%-100%), 524 of 524 infants treated for ROP (sensitivity, 100%; 95% CI, 99.3%-100%), 466 of 472 infants with type 2 ROP (sensitivity, 98.7%; 95% CI, 97.3%-99.4%), and 925 of 931 infants with type 1 or 2 ROP (sensitivity, 99.4%; 95% CI, 98.6%-99.7%). Of 7483 infants in the study, 2269 infants (30.3%) did not meet any of the criteria and would not have received examinations if the study hospitals had been using only these criteria for ROP screening. The currently recommended ROP screening criteria (BW < 1501g or GA ≤ 30 weeks) correctly predicted 456 of 459 infants with type 1 ROP (sensitivity, 99.3%; 95% CI, 98.1%-99.8%) and 470 of 472 infants with type 2 ROP (sensitivity, 99.6%; 95% CI, 98.5%-99.8%).

Figure 2. Modified Screening Criteria Developed Using Data From the Postnatal Growth and Retinopathy of Prematurity Study.

The criteria would be applied by beginning at the lower left hand of the diagram and proceeding in a clockwise fashion around the 6 criteria. If the gestational age (GA) is younger than 28 weeks, then the infant would receive retinal examinations. If the gestational age is 28 weeks or older, the next criterion (birth weight [BW]) would be checked, and so forth. If none of the criteria apply, then the infant would not receive retinal examinations. These criteria should not be used clinically until validated.

Discussion

We developed evidence-based ROP screening criteria using data from a cohort of at-risk infants. The cohort was representative of infants who received ROP examinations in North America and was diverse with regards to race/ethnicity, geography, and neonatal intensive care unit setting.²⁸ The criteria are relatively simple to use. They take a structure familiar to clinicians (BW, GA, and weight gain thresholds), use routinely collected data, and require minimal calculation, so they would have a minimal impact on workflow in the neonatal intensive care unit. Infants who meet either the BW criterion or GA criterion do not require weight gain calculations, because infants only need to meet 1 criterion to receive examinations. For larger-BW and older-GA infants, weight gain calculations involve absolute weight gain rather than weight gain rate, during 3 discrete periods. Including hydrocephalus as a criterion necessitates defining an upper limit of GA or BW for using the criterion, because not all premature newborns with hydrocephalus are at risk for ROP. One possibility is to apply the G-ROP criteria to all infants with a GA of 32 weeks or younger, because the oldest GA of an infant who developed type 1 or 2 ROP in the cohort was 32 weeks.

The G-ROP criteria correctly predicted the ROP status of all infants who developed type 1 ROP and all treated infants in the cohort. The sensitivity for predicting the development of type 1 ROP was 100%, higher than the sensitivity of the currently recommended screening thresholds (BW < 1501 g, GA < 30 weeks), which was 99.3%. Our new criteria missed a small percentage (1.3%) of type 2 ROP cases. However, currently recommended BW and GA screening thresholds would also have missed some cases of type 2 ROP. Moreover, treatment is not recommended for type 2 ROP.¹ It may be acceptable to not detect a small number of ROP cases that do not require treatment to spare many infants from having to undergo examinations, particularly if all very premature infants undergo ophthalmological examinations when they are older (eg, age 12 months) to detect long-term visual complications that are associated with prematurity, such as strabismus and high refractive errors.^34,35,36 Finally, outliers may be inevitable in a sufficiently large cohort of infants. Currently, such outliers are handled by including an additional ROP screening criterion: a poor postnatal course as determined by the neonatologist. If outliers are identified in further studies, then this type of additional criterion could be used alongside the G-ROP criteria for incorporating clinical judgment in screening decisions. However, in contrast to current screening guidelines, the G-ROP criteria alone correctly predicted all type 1 cases in this development cohort without the need for this additional criterion.

Despite their high sensitivity in this development study, the G-ROP criteria should not be used clinically until they are validated. The use of a clinical prediction model involves development, validation, and impact studies.^25,26,27,30 An important strength of the G-ROP criteria is that they were developed in a very large cohort that was broadly representative of infants undergoing ROP examinations in North America. Prior postnatal weight gain predictive models were developed using cohorts with small numbers of severe ROP cases (WINROP, 13 infants³²; PINT ROP, 67 infants¹⁵; CHOP ROP, 48 infants¹⁷; ROPScore, 24 infants¹⁶; and CO-ROP, 45 infants¹⁸), resulting in overfitting and decreased sensitivity in new, larger cohorts of infants.^13,19,24 In contrast, the G-ROP study cohort included 931 infants with at least type 2 ROP, of whom 459 (49.3%) developed type 1 ROP requiring treatment. This large number of outcome cases helps to minimize the potential for overfitting to the data and provides precise estimates of sensitivity.²⁵ Nevertheless, a predictive model should be validated in new patients to confirm its performance and determine its generalizability before clinical implementation.^26,27 Differences in medical characteristics between the G-ROP cohort and subsequent groups of infants may result in a decrease in sensitivity for type 1 ROP.^25,26,27 In particular, changes in neonatal care practices in the United States and Canada, such as shifts in oxygen saturation target ranges,³⁷ may affect the characteristics of infants who develop severe ROP and affect the performance of any model. If such changes in performance are identified during validation, the criteria should be updated based on a combined development and validation cohort.^26,27,30

The G-ROP criteria are unlikely to be generalizable to medical settings where higher-BW and older-GA infants regularly develop severe ROP, such as countries with developing neonatal care systems. In such settings, high oxygen use is likely to play a more central role in the pathogenesis of ROP because endogenous IGF-1 production is not deficient in older-GA infants,³⁸ making postnatal weight gain a less reliable predictor of severe ROP.^39,40

Limitations

There are additional potential limitations to consider. Retrospective data collection can introduce bias into the study design, although steps were taken to minimize such bias. We performed a feasibility study beforehand to identify data that were consistently documented and possible to collect retrospectively without inference. We took data quality measures, including a rigorous data collector certification process and extensive procedures to identify and correct data collection errors.²⁸ Retinal examinations and weight measurements were not performed in a regimented, a priori fashion for the G-ROP Study. However, the examinations were completed by ophthalmologists with expertise in ROP using standardized International Classification of ROP terms.⁴¹ Moreover, both the retinal examinations and the weight measurements reflect the regular variations that are seen in clinical practice among practitioners and across sites. Using such “real-life” data should result in more generalizable screening criteria.

Conclusions

The development of the G-ROP criteria brings us one step closer to incorporating slow postnatal weight gain into ROP screening. The criteria predict the development of severe ROP with a greater specificity and greater sensitivity than current ROP screening criteria; their use would have reduced the number of infants who required examinations in the development cohort by almost one-third while better capturing outlying high-BW, older-GA infants who developed type 1 ROP. If the criteria are validated and consensus in the ophthalmology and neonatology communities can be reached on the minimum performance standards for ROP screening, then changes to recommended ROP screening guidelines could be proposed.