Skip to main content
PMC home page
JAMA Network logoLink to JAMA Network
. 2019 Jul 3;137(9):1022–1028. doi: 10.1001/jamaophthalmol.2019.2433

Monitoring Disease Progression With a Quantitative Severity Scale for Retinopathy of Prematurity Using Deep Learning

Stanford Taylor 1, James M Brown 2, Kishan Gupta 1, J Peter Campbell 1, Susan Ostmo 1, R V Paul Chan 3, Jennifer Dy 4, Deniz Erdogmus 4, Stratis Ioannidis 4, Sang J Kim 1,5, Jayashree Kalpathy-Cramer 2,6, Michael F Chiang 1,7,✉, for the Imaging and Informatics in Retinopathy of Prematurity Consortium
PMCID: PMC6613341  PMID: 31268518

This study describes a quantitative retinopathy of prematurity severity score derived using a deep learning algorithm designed to evaluate plus disease and assess its utility for objectively monitoring retinopathy of prematurity progression.

Key Points

Question

Can a quantitative measurement of retinopathy of prematurity severity be tracked over time to identify disease progression?

Findings

In this cohort study of 871 infants with 5255 clinical examinations, a quantitative retinopathy of prematurity vascular severity score developed using an automated deep learning–based plus disease algorithm identified differences in the mean severity of eyes progressing to treatment-requiring retinopathy of prematurity compared with eyes that did not require treatment using only a posterior pole photograph.

Meaning

Tracking quantitative measurements of retinopathy of prematurity severity may be an effective method of identifying patients at risk for disease progression and in need of future retinopathy of prematurity treatment.

Abstract

Importance

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide, but clinical diagnosis is subjective and qualitative.

Objective

To describe a quantitative ROP severity score derived using a deep learning algorithm designed to evaluate plus disease and to assess its utility for objectively monitoring ROP progression.

Design, Setting, and Participants

This retrospective cohort study included images from 5255 clinical examinations of 871 premature infants who met the ROP screening criteria of the Imaging and Informatics in ROP (i-ROP) Consortium, which comprises 9 tertiary care centers in North America, from July 1, 2011, to December 31, 2016. Data analysis was performed from July 2017 to May 2018.

Exposure

A deep learning algorithm was used to assign a continuous ROP vascular severity score from 1 (most normal) to 9 (most severe) at each examination based on a single posterior photograph compared with a reference standard diagnosis (RSD) simplified into 4 categories: no ROP, mild ROP, type 2 ROP or pre-plus disease, or type 1 ROP. Disease course was assessed longitudinally across multiple examinations for all patients.

Main Outcomes and Measures

Mean ROP vascular severity score progression over time compared with the RSD.

Results

A total of 5255 clinical examinations from 871 infants (mean [SD] gestational age, 27.0 [2.0] weeks; 493 [56.6%] male; mean [SD] birth weight, 949 [271] g) were analyzed. The median severity scores for each category were as follows: 1.1 (interquartile range [IQR], 1.0-1.5) (no ROP), 1.5 (IQR, 1.1-3.4) (mild ROP), 4.6 (IQR, 2.4-5.3) (type 2 and pre-plus), and 7.5 (IQR, 5.0-8.7) (treatment-requiring ROP) (P < .001). When the long-term differences in the median severity scores across time between the eyes progressing to treatment and those who did not eventually require treatment were compared, the median score was higher in the treatment group by 0.06 at 30 to 32 weeks, 0.75 at 32 to 34 weeks, 3.56 at 34 to 36 weeks, 3.71 at 36 to 38 weeks, and 3.24 at 38 to 40 weeks postmenstrual age (P < .001 for all comparisons).

Conclusions and Relevance

The findings suggest that the proposed ROP vascular severity score is associated with category of disease at a given point in time and clinical progression of ROP in premature infants. Automated image analysis may be used to quantify clinical disease progression and identify infants at high risk for eventually developing treatment-requiring ROP. This finding has implications for quality and delivery of ROP care and for future approaches to disease classification.

Introduction

Early identification and management of retinopathy of prematurity (ROP) in infants born with low birth weights lead to improved long-term visual prognosis.1,2,3 The American Academy of Pediatrics, American Academy of Ophthalmology, and American Association for Pediatric Ophthalmology and Strabismus have established guidelines for ROP screening in preterm infants that rely heavily on data from the Early Treatment for ROP (ETROP) study and the Cryotherapy for ROP (CRYO-ROP) study and on nomenclature from the International Classification of ROP (ICROP).1,2,3,4,5 These data suggest that appropriate screening criteria (with high sensitivity for severe ROP) and timely intervention can greatly reduce the incidence of blindness. Current guidelines allow bedside ophthalmoscopic screening as well as telemedical screening using digital fundus imaging and remote interpretation, with follow-up and management based on ophthalmoscopic or image-based disease classification.5

Retinopathy of prematurity evolves on a relatively predictable timeline, with the level of severity progressing along a clinical continuum from mild to severe disease.1,2,3,6 On the basis of the ETROP trial, treatment of type 1 disease is known to improve visual outcomes compared with treatment at the level of CRYO-ROP threshold disease, but there was no benefit for treatment of less severe type 2 disease.2

However, ROP diagnosis is subjective and qualitative. Therefore, disease classification for the same infant may vary significantly among examiners,7,8,9,10,11,12 which may lead to real-world treatment differences.2,7 With the exception of zone I, stage 3 disease, the only difference between type 1 and type 2 disease (and thus the need for treatment) is the diagnosis of plus disease, which according to ICROP is an assessment of the arterial tortuosity and venous dilation of the retinal vessels compared with a historical photograph.4 The Benefits of Oxygen Saturation Targeting II trial found evidence of differences in type 1 ROP diagnosis among investigators in the trial based primarily on the difference in plus disease diagnosis.13 The lack of objective metrics of ROP disease severity and the presence of this same bias in ETROP and CRYO-ROP limit our ability to understand whether these differences in treatment severity may explain differences in visual outcomes or rates of retinal detachment among centers in the real world.

Recent advances in deep learning (DL) have demonstrated promise for automated and objective diagnosis of plus disease in ROP.14,15,16 Brown et al14 found that a fully automated plus disease classifier developed on data from the Imaging and Informatics in ROP (i-ROP) study, called i-ROP DL, was able to classify plus disease on a 3-level scale (no, pre-plus, and plus) with comparable or better accuracy than clinical experts. As a convolutional neural network, the i-ROP DL system generates hidden features for classifying plus disease based on a single image that presumably correlate with clinical disease features of arterial tortuosity and venous dilation but that are hidden in the deep network. Subsequent work by Redd et al16 found that a quantitative ROP vascular severity score developed using the i-ROP DL system correlated with ICROP disease severity in a cross-sectional evaluation of the i-ROP screening examinations and demonstrated promise as a screening tool in an ROP screening program. However, this technology has not been applied on a large scale to determine whether a continuous vascular severity score might be able to complement telemedicine or ophthalmoscopic disease screening to improve the objectivity of ROP screening, diagnosis, and management.15 In this study, we addressed this gap in knowledge by retrospectively applying this ROP vascular severity score to the prospective i-ROP cohort data to understand whether this score might be used to track changes over time in disease severity and identify patients at risk of progressing to severe treatment-requiring ROP (TR-ROP).

Methods

The i-ROP consortium is composed of 9 tertiary referral centers that screen high volumes of at-risk infants for ROP. The present analysis was approved by the Oregon Health & Science University Institutional Review Board, and the overall study was approved by each of the 9 participating institutions (Columbia University, Cornell University, University of Illinois at Chicago, William Beaumont Hospital, Children’s Hospital Los Angeles, Cedars-Sinai Medical Center, University of Miami, and Asociacion para Evitar la Ceguera en Mexico). All institutions abided by the tenets of the Declaration of Helsinki,17 and written informed consent was obtained from parents of all infants enrolled.

Data Set

Deidentified images from clinical examinations performed from July 1, 2011, to December 31, 2016, were reviewed. All images were obtained using a commercially available camera (RetCam, Natus Medical Inc). Each study eye examination was assigned a reference standard diagnosis (RSD) including all categories of ROP classification: zone (I-III), stage (1-5), and plus (plus, pre-plus, or no plus). The RSD was based on combined findings from ophthalmoscopic and image-based examinations by multiple experts using methods previously published.18 On the basis of the ICROP classification, the examinations were grouped into disease subcategories for each eye examination (by eye): no ROP, mild ROP, type 2 ROP or pre-plus, or type 1 ROP. The cohort was split into 2 groups for this analysis: eyes that developed TR-ROP and eyes that did not. In most cases, eyes in the TR-ROP group had type 1 ROP; however, previous analysis of this data set demonstrated that approximately 10% of the time, patients were treated for less than type 1 ROP based on clinical judgment of the examining ophthalmologist.19 Images were excluded if 2 of 3 image graders labeled them unacceptable for diagnosis or if the clinical diagnosis was stage 4 or 5 ROP with retinal detachment. Data analysis was performed from July 2017 to May 2018.

Image Analysis: DL and Quantitative Score

The i-ROP DL system was used to analyze each image in the data set and classify the probability of plus disease, pre-plus disease, and no disease using previously published methods based on a single posterior-pole fundus image.14 On the basis of these 3 probabilities, an automated ROP vascular severity score was then assigned to each image from 1 (normal retinal vasculature) to 9 (worst retinal vasculature, eg, severe plus disease) using methods previously described ([1 × probability of no disease] + [5 × probability of pre-plus disease] + [9 × probability of plus disease]).16 The disease course was then assessed between the 2 groups throughout the period of inpatient ROP screening by grouping the examinations into five 2-week intervals between 30 and 40 weeks of postmenstrual age (PMA).

Statistical Analysis

Changes in ROP vascular severity score were calculated by looking at the 4 intervals among these 5 time points. Multivariable logistic regression was performed to identify whether early change (before 35 weeks PMA) in the ROP vascular severity score was independently associated with the future development of TR-ROP (with P < .05 denoting statistical significance). Statistical analysis was performed using MATLAB 2014A (MathWorks). Descriptive statistics, 2-tailed, unpaired t test, Kruskal-Wallis test, Wilcoxon rank sum test with Bonferroni correction, and 2-way analysis of variance were used to analyze for an association between the ROP vascular severity score and ROP progression.

Results

Description of Study Cohort

A total of 5255 clinical examinations from 871 infants (mean [SD] gestational age, 27.0 [2.0] weeks; 493 [56.6%] male; mean [SD] birth weight, 949 [271] g) were analyzed. In total, 91 eyes (5.4% of the cohort) progressed to TR-ROP. The mean (SD) gestational age was 25.1 (1.2) weeks for those who required treatment and 27.2 (2.0) weeks for those who did not require treatment (P < .001 for both comparisons). The mean (SD) birth weight was 682 (195) g in those who required treatment compared with 964 (297) g in those who did not (P < .001). Treatment occurred a mean (SD) of 37.7 (3.1) weeks PMA. Figure 1 and Figure 2 display a representative example demonstrating the progression of images, RSD classifications, and ROP vascular severity score over time for 2 patients: 1 who developed TR-ROP and 1 who did not.

Figure 1. Serial Wide-Angle Retinal Images and Corresponding Quantitative Vascular Severity Score From Deep Learning in an Infant Who Did Not Require Treatment.

Figure 1.

Open in a new tab

Images are from an infant with a gestational age of 260/7 weeks and a birth weight of 830 g. The retinopathy of prematurity vascular severity score (range, 1-9) was 1.04 in A, 1.44 in B, and 1.36 in C.

Figure 2. Serial Wide-Angle Retinal Images and Corresponding Quantitative Vascular Severity Score From Deep Learning in Infant Who Required Treatment.

Figure 2.

Open in a new tab

Images are from an infant with a gestational age of 260/7 weeks and a birth weight of 783 g. The retinopathy of prematurity vascular severity score (range, 1-9) was 1.04 in A, 5.00 in B, and 8.64 in C.

Comparison of Vascular Severity Score vs RSD

Figure 3 shows the distribution of all the ROP vascular severity scores for the entire population of patients (cross-sectionally) compared with the RSD disease classification. The median ROP vascular severity of score of examinations with an RSD of no ROP was 1.1 (IQR, 1.0-1.5) compared with 1.5 (IQR, 1.1-3.4) for eyes with mild ROP, 4.6 (IQR, 2.4-5.3) for eyes with type 2 ROP and/or pre-plus disease, and 7.5 (IQR, 5.0-8.7) for eyes with TR-ROP (P < .001 for differences in ROP severity scores between all categorizations by Wilcoxon rank sum test with Bonferroni correction).

Figure 3. Box Plots of Retinopathy of Prematurity (ROP) Vascular Severity Score Based on Reference Standard Diagnosis (RSD) in International Classification of ROP Categories.

Figure 3.

Open in a new tab

P < .001 by Wilcoxon rank sum test with Bonferroni correction. Dots indicate medians; and boxes, IQRs.

Vascular Severity Score for Monitoring Disease Progression

Figure 4 displays the distribution of ROP vascular severity scores in cohorts that eventually progressed to TR-ROP vs those who did not. Overall, population-level differences between the 2 cohorts were seen early in follow-up and increased progressively over time. When the long-term differences in the median severity scores across time between the eyes requiring treatment and those that did not eventually require treatment were compared, the median score was higher in the treatment group by 0.06 at 30 to 32 weeks, 0.75 at 32 to 34 weeks, 3.56 at 34 to 36 weeks, 3.71 at 36 to 38 weeks, and 3.24 at 38 to 40 weeks postmenstrual age (P < .001 for all comparisons). The median vascular severity score increased with PMA even in patients who did not progress to TR-ROP, with a higher median score of 1.87 at 38 to 40 weeks PMA compared with a median score of 1.02 at 30 to 32 weeks PMA (P < .001). This finding suggests that PMA should be factored into any image-based risk prediction models and that the convolutional neural network may be associated with PMA based on vascular features.

Figure 4. Violin Plot of Distribution of Retinopathy of Prematurity (ROP) Vascular Severity Scores in Eyes That Never Required Treatment Compared With Eyes That Developed Treatment-Requiring Disease .

Figure 4.

Open in a new tab

P < .001 for all comparisons.

Risk Modeling

Figure 5 shows the rate of change of the ROP vascular severity score over time. In this analysis, we found that eyes that developed TR-ROP had a higher mean rate of change in the ROP vascular severity score (range, 0.40-1.32 points per week) compared with eyes that never developed TR-ROP (range, 0.08-0.20 point per week). The early separation of vascular severity scores between the 2 cohorts suggests that this information may be useful for determining which eyes may be at highest risk for progression to TR-ROP. In multivariable logistic regression controlling for birth weight and gestational age, the rate of change of the ROP vascular severity score at both 32 and 34 weeks was independently associated with future TR-ROP.

Figure 5. Mean (SD) Rate of Retinopathy of Prematurity (ROP) Vascular Severity Score Change per Week by Postmenstrual Age in Eyes That Never Required Treatment Compared With Eyes That Developed Treatment-Requiring Disease .

Figure 5.

Open in a new tab

Controlling for birth weight and gestational age, multivariable logistic regression found the rate of change in the ROP vascular severity score was independently associated with future treatment. Error bars indicate SDs.

Discussion

We report the results of an automated ROP vascular severity score applied to the i-ROP multi-institution cohort study. The key findings of this study are as follows. First, a quantitative ROP vascular severity score may be derived from DL methods and was associated with clinical disease severity. Second, with the use of this technique, ROP severity can be tracked over time and used to identify patients with TR-ROP. Third, the rate of change in the ROP vascular severity score may be independently associated with disease worsening and have implications for modeling future disease risk.

Brown et al14 found that DL methods can classify plus disease with comparable or better accuracy than experts. The first key finding of this study suggests that we may extend that work to demonstrate that an ROP vascular severity score calculated from the DL algorithm can quantify severity differences among clinical disease levels (Figure 3). This has important implications for introducing an element of objectivity into the diagnosis of ROP, which currently remains based on the subjective interpretation of qualitative disease features. This finding has implications for research, teaching, and patient care. Heterogeneity in plus disease diagnosis has existed in, to our knowledge, all the randomized clinical trials in ROP, which limits generalizability of the findings to specific examiners and may account for heterogeneity in outcomes.1,2,3,13 Use of this severity score in clinical trials (or applied retrospectively to existing clinical trials) could provide insight into the current practice patterns of treating physicians as they relate to clinical trial outcomes. A better understanding of the objective level of disease severity being labeled as plus disease will lead to more standardized application of treatment guidelines by trainees in the future. There may be other states, such as pulmonary hypertension or the use of nitrous oxide, which elevate the ROP vascular severity score. In the future, incorporating an objective severity score into the evaluation of patients with ROP could, like other objective measurements of disease severity (eg, blood pressure), be put into the clinical context for decision making.

Furthermore, there are multiple lines of evidence indicating that regional variation exists in the application of clinical trial data in ROP.2,7,9,10,11,13,20 Specifically, because of the subjectivity of plus disease diagnosis and evidence of systematic bias between examiners, infants with the same image-graded level of disease may be treated differently by different physicians.13 One future application of the ROP vascular severity score would be to prospectively analyze objective treatment thresholds to identify the appropriate level of plus disease to guide treatment. In this study, the mean score at the time of treatment was 7. Gupta et al21 have reported that a higher ROP vascular severity score at the time of treatment was associated with a risk of laser treatment failure (needing >1 laser treatment). Future work may identify whether management should be guided by an objective level of vascular severity (an objective metric of plus disease) or whether other factors, such as the extent of extraretinal neovascularization (as was previously incorporated in the CRYO-ROP threshold disease definition), should return to the management paradigms.

The second key study finding is that ROP vascular severity score may be used to track clinical disease progression. Although there have been multiple reports of using computer-based image analysis to classify disease at a specific point in time, few articles15,22,23 have looked at objectively monitored changes in disease severity over time. We found that the ROP vascular severity score increased in all infants during development (Figure 4). These quantitative methods may provide insights about the normal process of vascularization and identify infants at risk for developing more aggressive disease. The severity score of infants who would subsequently develop TR-ROP was different at every time point, with differences emerging by 34 weeks PMA (Figure 4). This finding suggests that information contained in the image at a specific point in time may identify patients at higher future risk for developing severe TR-ROP. This finding may have important clinical applications for prognosis, delivery of care, and disease management using traditional bedside examination or telemedicine.5,6,24,25

The third key finding is that changes in quantitative ROP vascular severity score between images over time may help identify patients who eventually require treatment for ROP. This finding appears to be an independent risk factor beyond clinical factors; demographic factors, such as birth weight and gestational age; and ROP vascular severity score at individual points in time. Although some work has considered how practitioners may or may not factor in change in retinal vascular appearance over time,22,23 this clinical judgment is not part of existing classification schemas or educational paradigms. Moreover, current screening paradigms do not factor this information into referral, treatment, or follow-up guidelines.5 Further analyses of these and other data using DL may increase understanding of the natural progression of ROP and clinical phenotypic characteristics associated with disease progression. A future randomized clinical trial could then assess benefit of earlier intervention compared with ETROP type 1 disease.

There have been a number of attempts to develop risk models in ROP, with 2 main objectives: (1) to reduce the number of screening examinations by identifying patients at lowest risk of having disease and (2) to identify patients at highest risk of progressing to treatment.26,27,28,29,30 Redd et al16 demonstrated that applied retrospectively to the i-ROP cohort, use of this ROP vascular severity score may be associated with a reduction in the burden of ophthalmoscopic examinations by 80% with an area under the receiver operating characteristic of 0.95 for detection of type 1 disease. The ETROP trial also used the Risk Management Model for ROP 2 (RM-ROP2) to randomize only those infants at highest risk of progression to earlier treatment.29 In this study, the mean rate of change in ROP vascular severity score was higher in eyes progressing to TR-ROP as early as 32 weeks (ie, between the first and second screening examinations), which is 5 weeks before the mean PMA of treatment in this cohort. These pretreatment differences in rate of change between the 2 groups were significant at all time points analyzed and were also independent of birth weight or gestational age at earlier PMAs. These data suggest that the rate of change of the ROP vascular severity score may provide added prognostic value to the clinical demographic factors included in most risk models, which could lead to closer follow-up for high-risk infants and provide an objective measurement of disease risk progression to guide clinical decision making.

Limitations

There are a number of limitations and opportunities for future improvements in this analysis. The study was cross-sectional and separated into only 2 cohorts. In the future, it would be beneficial to better understand change in DL features between individual patients over time and in patients who develop mild and moderate disease vs no ROP. The technology is currently only available in research studies under institutional review board approval; thus, the real-world consequences of these findings are currently limited. Further work needs to be performed to demonstrate the association between image quality, field of view, and camera system with the ROP severity score. Finally, this study was performed in a North American cohort of infants. It will be necessary to validate this system in larger patient cohorts from developed countries. Furthermore, it will be necessary to perform similar analysis in cohorts outside North America to assess the generalizability of the study findings, especially in low- and middle-income countries where ROP is epidemic and oxygen regulation is known to produce a more aggressive phenotype.

Conclusions

Deep learning using convolutional neural networks is a rapidly developing method of automated image analysis in a variety of medical specialties, including ophthalmology.31,32,33 However, the precise way in which this technology will be incorporated into existing practice paradigms or change existing practice paradigms remains to be seen.34 In this study, an automated ROP vascular severity score obtained from posterior-pole fundus images in patients with ROP effectively distinguished disease progression in infants undergoing screening for ROP. With further validation, these results may have broad implications for the potential role of similar technologies in ROP screening, diagnosis, and management.

References

  • 1.Cryotherapy for Retinopathy of Prematurity Cooperative Group Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results. Arch Ophthalmol. 1988;106(4):471-479. doi: 10.1001/archopht.1988.01060130517027 [DOI] [PubMed] [Google Scholar]
  • 2.Reynolds JD, Dobson V, Quinn GE, et al. ; CRYO-ROP and LIGHT-ROP Cooperative Study Groups . Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120(11):1470-1476. doi: 10.1001/archopht.120.11.1470 [DOI] [PubMed] [Google Scholar]
  • 3.Early Treatment for Retinopathy of Prematurity Cooperative Group Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121(12):1684-1694. doi: 10.1001/archopht.121.12.1684 [DOI] [PubMed] [Google Scholar]
  • 4.International Committee for the Classification of Retinopathy of Prematurity The International Classification of Retinopathy of Prematurity Revisited. Arch Ophthalmol. 2005;123(7):991-999. doi: 10.1001/archopht.123.7.991. [DOI] [PubMed]
  • 5.Fierson WM; American Academy of Pediatrics Section on Ophthalmology; American Academy of Ophthalmology; American Association for Pediatric Ophthalmology and Strabismus; American Association of Certified Orthoptists . Screening examination of premature infants for retinopathy of prematurity. Pediatrics. 2018;142(6):e20183061. doi: 10.1542/peds.2018-3061 [DOI] [PubMed] [Google Scholar]
  • 6.Quinn GE, Ying G-S, Daniel E, et al. ; e-ROP Cooperative Group . Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity. JAMA Ophthalmol. 2014;132(10):1178-1184. doi: 10.1001/jamaophthalmol.2014.1604 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Campbell JP, Ryan MC, Lore E, et al. ; Imaging & Informatics in Retinopathy of Prematurity Research Consortium . Diagnostic discrepancies in retinopathy of prematurity classification. Ophthalmology. 2016;123(8):1795-1801. doi: 10.1016/j.ophtha.2016.04.035 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Quinn GE, Ells A, Capone A Jr, et al. ; e-ROP (Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity) Cooperative Group . Analysis of discrepancy between diagnostic clinical examination findings and corresponding evaluation of digital images in the telemedicine approaches to evaluating acute-phase retinopathy of prematurity study. JAMA Ophthalmol. 2016;134(11):1263-1270. doi: 10.1001/jamaophthalmol.2016.3502 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Campbell JP, Kalpathy-Cramer J, Erdogmus D, et al. ; Imaging and Informatics in Retinopathy of Prematurity Research Consortium . Plus disease in retinopathy of prematurity: a continuous spectrum of vascular abnormality as a basis of diagnostic variability. Ophthalmology. 2016;123(11):2338-2344. doi: 10.1016/j.ophtha.2016.07.026 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Kalpathy-Cramer J, Campbell JP, Erdogmus D, et al. ; Imaging and Informatics in Retinopathy of Prematurity Research Consortium . Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis. Ophthalmology. 2016;123(11):2345-2351. doi: 10.1016/j.ophtha.2016.07.020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Campbell JP, Ataer-Cansizoglu E, Bolon-Canedo V, et al. ; Imaging and Informatics in ROP (i-ROP) Research Consortium . Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis. JAMA Ophthalmol. 2016;134(6):651-657. doi: 10.1001/jamaophthalmol.2016.0611 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Chiang MF, Thyparampil PJ, Rabinowitz D. Interexpert agreement in the identification of macular location in infants at risk for retinopathy of prematurity. Arch Ophthalmol. 2010;128(9):1153-1159. doi: 10.1001/archophthalmol.2010.199 [DOI] [PubMed] [Google Scholar]
  • 13.Fleck BW, Williams C, Juszczak E, et al. . An international comparison of retinopathy of prematurity grading performance within the Benefits of Oxygen Saturation Targeting II trials. Eye (Lond). 2017;123:1-7. doi: 10.1038/eye.2017.150 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Brown JM, Campbell JP, Beers A, et al. ; Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium . Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmol. 2018;136(7):803-810. doi: 10.1001/jamaophthalmol.2018.1934 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Brown JM, Kalpathy-Cramer J, Campbell JP, et al. Fully automated disease severity assessment and treatment monitoring in retinopathy of prematurity using deep learning. In: Proceedings of Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications; March 6, 2018; Houston, Texas.
  • 16.Redd TK, Campbell JP, Brown JM, et al. . Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity [published online November 23, 2018]. Br J Ophthalmol. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.World Medical Association World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191. doi: 10.1001/jama.2013.281053 [DOI] [PubMed] [Google Scholar]
  • 18.Ryan MC, Ostmo S, Jonas K, et al. . Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology. AMIA Annu Symp Proc. 2014;2014:1902-1910. [PMC free article] [PubMed] [Google Scholar]
  • 19.Gupta MP, Chan RVP, Anzures R, Ostmo S, Jonas K, Chiang MF; Imaging and Informatics in ROP Research Consortium . Practice patterns in retinopathy of prematurity treatment for disease milder than recommended by guidelines. Am J Ophthalmol. 2016;163:1-10. doi: 10.1016/j.ajo.2015.12.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875-880. doi: 10.1001/archopht.125.7.875 [DOI] [PubMed] [Google Scholar]
  • 21.Taylor S, Campbell JP, Gupta K, et al. . A quantitative severity scale for retinopathy of prematurity using deep learning to monitor disease regression after treatment [published online July 3, 2019]. JAMA Ophthalmol. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Wallace DK, Freedman SF, Zhao Z. Evolution of plus disease in retinopathy of prematurity: quantification by ROPtool. Trans Am Ophthalmol Soc. 2009;107:47-52. [PMC free article] [PubMed] [Google Scholar]
  • 23.Thyparampil PJ, Park Y, Martinez-Perez ME, et al. . Plus disease in retinopathy of prematurity: quantitative analysis of vascular change. Am J Ophthalmol. 2010;150(4):468-475.e2. doi: 10.1016/j.ajo.2010.04.027 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Chiang MF, Melia M, Buffenn AN, et al. . Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography: a report by the American Academy of Ophthalmology. Ophthalmology. 2012;119(6):1272-1280. doi: 10.1016/j.ophtha.2012.01.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Weaver DT. Telemedicine for retinopathy of prematurity. Curr Opin Ophthalmol. 2013;24(5):425-431. doi: 10.1097/ICU.0b013e3283645b41 [DOI] [PubMed] [Google Scholar]
  • 26.Jung JL, Wagner BD, McCourt EA, et al. . Validation of WINROP for detecting retinopathy of prematurity in a North American cohort of preterm infants. J AAPOS. 2017;21(3):229-233. doi: 10.1016/j.jaapos.2017.05.004 [DOI] [PubMed] [Google Scholar]
  • 27.Binenbaum G, Ying G-S, Tomlinson LA; Postnatal Growth and Retinopathy of Prematurity (G-ROP) Study Group . Validation of the Children’s Hospital of Philadelphia Retinopathy of Prematurity (CHOP ROP) Model. JAMA Ophthalmol. 2017;135(8):871-877. doi: 10.1001/jamaophthalmol.2017.2295 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.McCourt EA, Ying G-S, Lynch AM, et al. ; G-ROP Study Group . Validation of the Colorado Retinopathy of Prematurity Screening Model. JAMA Ophthalmol. 2018;136(4):409-416. doi: 10.1001/jamaophthalmol.2018.0376 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Onofrey CB, Feuer WJ, Flynn JT. The outcome of retinopathy of prematurity: screening for retinopathy of prematurity using an outcome predictive program. Ophthalmology. 2001;108(1):27-34. doi: 10.1016/S0161-6420(00)00436-X [DOI] [PubMed] [Google Scholar]
  • 30.Kim SJ, Port AD, Swan R, Campbell JP, Chan RVP, Chiang MF. Retinopathy of prematurity: a review of risk factors and their clinical significance. Surv Ophthalmol. 2018;63(5):618-637. doi: 10.1016/j.survophthal.2018.04.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Gulshan V, Peng L, Coram M, et al. . Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316(22):2402-2410. doi: 10.1001/jama.2016.17216 [DOI] [PubMed] [Google Scholar]
  • 32.Lee A, Taylor P, Kalpathy-Cramer J, Tufail A. Machine learning has arrived! Ophthalmology. 2017;124(12):1726-1728. doi: 10.1016/j.ophtha.2017.08.046 [DOI] [PubMed] [Google Scholar]
  • 33.Ting DSW, Pasquale LR, Peng L, et al. . Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. 2019;103(2):167-175. doi: 10.1136/bjophthalmol-2018-313173 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Ting DSW, Wu W-C, Toth C. Deep learning for retinopathy of prematurity screening [published online November 23, 2018]. Br J Ophthalmol. [DOI] [PubMed] [Google Scholar]

Articles from JAMA Ophthalmology are provided here courtesy of American Medical Association

RESOURCES