A Prediction Model for Retinopathy of Prematurity–Is It Ready for Prime Time?

Retinopathy of prematurity (ROP), the leading cause of preventable childhood blindness worldwide, is traditionally detected by eye examinations performed by ophthalmologists on infants at risk for ROP. Because of the low diagnostic yield of these examinations for identifying infants that require treatment ofROP, various statistical prediction models have been developed to identify infants at high ROP risk who require frequent eye examinations and infants at low risk who require less-frequent or no ROP examinations.¹ These prediction models use statistical modeling approaches of various complexity, usually including birth weight (BW), gestational age (GA), and postnatal factors, such as oxygen exposure or postnatal weight gain. The model performance is usually evaluated using sensitivity, specificity, or the reduction in number of infants examined for detecting the ROP outcome of interest (eg, severe ROP, ROP requiring treatment, type 1 ROP). Most prediction models were developed from a small numbers of infants who have the ROP outcomes of interest, potentially leading to over-fitting or optimistic estimates of model performance. When applying prediction models to an independent cohort through external validation, their performance usually becomes poorer. Because no prediction model works well universally, research on developing and validating robust ROP prediction models for clinical use continues to be of interest.

In this issue of JAMA Ophthalmology, Pivodic et al² developed and validated an individual risk prediction model (DIGIROP-Birth) for identifying ROP requiring treatment using purely birth characteristics (BW, GA, and sex), without consideration of any postnatal factors. The prediction model was developed based on 7286 infants born prematurely who were listed in the Swedish National Patient Registry and screened for ROP from 2007 to 2017. In Sweden, infants with GAs less than 31 weeks or severe illness were registered for ROP screening (with an approximate 97% coverage rate). In the model development cohort, the model had an area under receiver operating characteristic curve (AUC) of 0.90. When the model was externally validated in a new cohort in Sweden (n = 323), a US cohort (n = 1535), and an European cohort (n = 354), the AUC remained similarly high (0.94 for the Swedish cohort, 0.87 for the US cohort, and 0.90 for the European cohort). Furthermore, when the DIGIROP-Birth model was compared with a few existing prediction models that require postnatal factors (the Children’s Hospital of Philadelphia-ROP, Omaha-ROP, Colorado-ROP, and weight, insulinlike growth factor I, neonatal ROP models) in a US cohort, this model suggested higher specificity than other models at the same high sensitivity (≥96.8%). However, this model was not compared with a recently developed³ and externally validated Growth and Retinopathy of Prematurity (G-ROP) modified screening criteria.^4,5 The G-ROP screening, developed from a large representative cohort of 7483 infants at risk in North America, requires that infants undergo ROP examination if any of 6 criteria are met: (1) a GA smaller than 28 weeks, (2) a BW less than 1051 g, (3) weight gain less than 120 g during days of life 10 to 19, (4) weight gain less than 180 g during days of life 20 to 29, (5) weight gain less than 170 g during days of life 30 to 39, or (6) the presence of hydrocephalus. The G-ROP criteria resulted in the correct identification of all cases of type 1 ROP and reduced the number of infants who required ROP examinations by 30%.³ Similar performance outcomes were found in a large prospective validation cohort.⁵

There are several strengths of the DIGIROP-Birth model. First, this model was developed from a large cohort with a large number of infants requiring ROP treatment and was validated externally in several cohorts from developed countries. Second, to make the model accessible for clinical care, an easy-to-use risk calculator was available free of charge at https://www.digirop.com/. In this risk calculator, the input only requires the infant’s whole GA (in weeks) and the partial GA (in days), BW in grams, and sex. The output provides the cumulative risk and its 95% CI for an infant developing ROP that requires treatment by 20 weeks after birth. This numeric risk estimate is informative to clinicians and parents for decision-making on ROP examination. Finally, the model allows early identification of infants at high risk (right after birth), since no postnatal factors are required. This early use of the prediction model should be helpful for planning ROP examination and follow-up schedules. Early planning may improve compliance of ROP examinations and decrease blindness from ROP, because noncompliance of follow-up for ROP examination is a major cause of blindness from ROP, particularly in developing countries.⁶

Before applying this prediction model to a population of premature infants, some considerations should be made. First, regarding generalizability, the prediction model was developed using infants in Sweden with GAs less than 31 weeks. Because ROP screening criteria vary across countries, this model is applicable to infants with GAs less than 31 weeks and should not be applied to infants with GAs of 31 weeks or more. In the G-ROP study of7483 infants eligible for ROP screening in North America (with BW <1501 g, GA <32 weeks, or a poor postnatal course), 1440 infants (19.2%) had a GA of 31 weeks or more. Because most of these infants with larger GAs will not develop ROP that requires treatment, prediction models are particularly needed to identify the small subset of infants at high risk. Even among infants with GAs less than 31 weeks, this model developed from infants in Sweden with excellent neonatal care is unlikely to be generalizable to infants from countries with developing neonatal care systems. The biological mechanism for ROP development and progression may differ between high-income countries and low-income countries. In low-income countries, high oxygen use and poor neonatal care are likely to play central roles in the pathogenesis of ROP that can cause the ROP that requires treatment in infants with larger GAs. For example, the WINROP model, developed using BW, GA, and postnatal weight gain in infants in Sweden, was validated in Sweden and a few developed countries, but it worked poorly in infants of Mexico, particularly in infants with GAs larger than 32 weeks.¹ Second, postnatal factors play an important role in development of severe ROP,⁷ and ignoring postnatal factors may lead to errors in detection of some cases of ROP that require treatment and are attributed to the postnatal factors. Finally, although the DIGIROP-Birth model provides an individual risk estimate of developing ROP that requires treatment, the sensitivity, specificity, and optimal cut point for detecting ROP that requires treatment in the cohort in Sweden were not assessed yet. Without guidance on optimal cut points, users of this prediction model may be uncertain on how to use the estimated individual risk for decision-making on which infants need ROP examination.

Clinicians are now equipped with several ROP prediction models, but they are challenged on which to choose for clinical use. Candidate prediction models should be evaluated and compared in a population of interest before applying them for clinical use. Because G-ROP screening criteria and DIGIROP-Birth are developed from large cohorts (>7000 infants) and are externally validated, it seems reasonable to evaluate these models in populations of interest before adopting 1 or both models for clinical use. The model with the best performance (the highest sensitivity and the largest reduction in the ROP examinations) should be considered. The ease of use and amount of data required also should be considered if 2 models have similar performance.

Although ROP prediction models show great promise in reducing ROP examinations and improving the ROP management, research on prediction models is still needed before widespread application to clinical practice can occur. It will be important to test and validate various prediction models in clinical practice, comparing performance in clinical settings. Comparing G-ROP screening criteria and the DIGIROP-Birth prediction model for detecting ROP that requires treatment in rich G-ROP data sets or other large cohorts also should be of value. Optimal models hopefully will reduce blindness from ROP and increase the cost-effectiveness of ROP screening.