Skip to main content
NCBI home page
JAMA Network logoLink to JAMA Network
. 2018 Apr 26;136(6):648–655. doi: 10.1001/jamaophthalmol.2018.1195

Accuracy and Reliability of Eye-Based vs Quadrant-Based Diagnosis of Plus Disease in Retinopathy of Prematurity

Sang Jin Kim 1,2, J Peter Campbell 1, Jayashree Kalpathy-Cramer 3,4, Susan Ostmo 1, Karyn E Jonas 5, Dongseok Choi 1,6, R V Paul Chan 5,7, Michael F Chiang 1,8,✉, for the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium
PMCID: PMC6145779  PMID: 29710185

Key Points

Question

Is plus disease in retinopathy of prematurity more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of individual quadrants?

Findings

In this multicenter cohort study including 141 preterm infants, graders diagnosed 197 eyes by reviewing the entire wide-angle retinal images (eye-based diagnosis; plus vs not plus) and subsequently by reviewing each of the cropped quadrant images (plus vs not plus), which was combined into a quadrant-based diagnosis. Graders had lower intragrader and intergrader agreement and agreement with reference standard diagnosis using quadrant-based diagnosis than eye-based diagnosis.

Meaning

Eye-based diagnosis may have advantages over quadrant-based diagnosis regarding reliability and accuracy.

This multicenter cohort study compares eye-based vs quadrant-based diagnosis of plus disease in infants with retinopathy of prematurity and provides insight for ophthalmologists about the diagnostic process.

Abstract

Importance

Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants.

Objective

To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process.

Design, Setting, and Participants

In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017.

Interventions

Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc.

Main Outcomes and Measures

Intragrader and intergrader reliability (absolute agreement and κ statistic) and accuracy compared with the reference standard diagnosis.

Results

Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen κ range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss κ, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader.

Conclusions and Relevance

Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

Introduction

Retinopathy of prematurity (ROP) is a proliferative retinopathy affecting premature infants and is a major cause of childhood blindness worldwide.1,2 Development of an international classification system of ROP (ICROP) has provided the infrastructure for improving clinical care and supporting multicenter research trials in ROP.3,4,5,6,7,8 Among the diagnostic parameters, presence of plus disease has been found by the Cryotherapy for ROP5 and the Early Treatment for ROP6 studies to be the most critical finding that indicates potentially blinding disease that requires treatment.

Plus disease is defined as venous dilatation and arteriolar tortuosity within the posterior retinal vessels, which is greater than or equal to that of a standard published photograph selected by expert consensus during the 1980s.3 Beyond comparison with a standard photograph, the Cryotherapy for ROP5 and the Early Treatment for ROP6 trials did not specifically indicate how plus disease should be diagnosed. In 2000, the multicenter Supplemental Therapeutic Oxygen for Prethreshold ROP7 study defined that plus disease should be diagnosed if there was sufficient dilatation and tortuosity in at least 2 quadrants of the retina. In 2005, this definition requiring 2 or more quadrants of vascular abnormality was incorporated into the revised ICROP.4 However, to our knowledge, no specific guidance was provided regarding how to apply this definition toward actual clinical diagnosis.

Despite a standardized definition of plus disease, there is significant diagnostic variability even among experts in the diagnostic outcome as well as in the diagnostic process of plus disease.9,10,11,12,13,14,15,16,17,18,19,20 This is a major problem because the presence of plus disease is the key indicator for severe, treatment-requiring disease.6 Furthermore, experts often deviate from the published definition of plus disease by incorporating features into their diagnosis that are not part of the published definition.9,10,11,12,13,14,15,18,19,20

Both ophthalmoscopic and image-based examination provide information about all quadrants simultaneously. For that reason, it has never been clear whether plus disease diagnosis should be performed using quadrant-based (ie, diagnose each quadrant individually as plus, pre-plus, or normal and integrate findings into an overall diagnosis)11,21,22,23 or eye-based (ie, assess overall retinal appearance)12,17 methods. Furthermore, it is not clear which of these approaches is more accurate. The purpose of this study is to directly compare the intragrader reliability, intergrader reliability, and overall accuracy of eye-based vs quadrant-based diagnosis for plus disease.

Methods

This study was approved by the institutional review board at the coordinating center (Oregon Health and Science University, Portland) and at each of 8 study centers (Columbia University, New York, New York; University of Illinois at Chicago; William Beaumont Hospital, Royal Oak, Michigan; Children’s Hospital Los Angeles, Los Angeles, California; Cedars-Sinai Medical Center, Los Angeles, California; University of Miami, Miami, Florida; Weill Cornell Medical Center, New York, New York; and Asociacion para Evitar la Ceguera en Mexico, Mexico City, Mexico). This study was conducted in accordance with the Declaration of Helsinki.24 Written informed consent for the study was obtained from parents of all infants enrolled.

Data Set

As part of the Imaging and Informatics in Retinopathy of Prematurity study, a multicenter cohort study, we developed a database of 197 wide-angle retinal images of the posterior retina from 141 preterm infants, which were taken using a wide-angle fundus camera (RetCam; Natus Medical Incorporated) between July 2011 and December 2016. The mean (SD) gestational age of included infants was 27.0 (2.6) weeks, and 65 (46.1%) were female. Of the 141 infants, 116 (82.3%) were white, 12 (8.5%) were African American, and 13 (9.2%) were other races/ethnicities. At each study center, infants underwent serial clinical examinations by a retinal specialist or pediatric ophthalmologist experienced in ROP. This was done by ophthalmoscopic examination at 8 study centers and a combination of telemedicine and ophthalmoscopy at 1 study center. All clinical examination findings were documented using ICROP criteria. A reference standard diagnosis (RSD) using ICROP criteria was assigned to each of the 197 images, as previously described.17,18,25 In brief, the RSD was established based on the consensus diagnosis that combined the image-based diagnosis (typically from 5 images: posterior, temporal, nasal, superior, and inferior retina) by 3 independent trained graders and the clinical diagnosis as described above. Among the 197 study images, 31 (15.7%) had an RSD of plus disease, 62 (31.5%) had an RSD of pre-plus disease, and 104 (52.8%) had an RSD of normal. Each of the 197 wide-angle retinal images was cropped into 4 quadrant images (superotemporal, inferotemporal, superonasal, and inferonasal) by dividing the posterior retinal images with vertical and horizontal lines bisecting the optic disc using image processing software (ImageJ; National Institutes of Health).

Image Grading

Six trained graders (4 ophthalmologists experienced in ROP and 2 nonphysicians experienced in review of ROP images) independently graded 197 images with eye-based diagnosis using a web-based system developed by the authors. Additional demographic information, such as gestational age, was not provided to graders. An eye-based diagnosis was defined as the diagnosis (plus, pre-plus, or normal) given after reviewing the entire image covering all 4 quadrants (Figure 1).

Figure 1. Workflow of Image Grading With Eye-Based Diagnosis.

Figure 1.

Open in a new tab

Subsequently, the 788 quadrant images were provided to the graders 1 at a time in random order (Figure 2). Graders were asked to diagnose the image as plus, pre-plus, or normal. A quadrant-based diagnosis of plus disease diagnosis was assigned when 2 or more quadrants of an image were independently diagnosed as having plus disease. The workflow of image grading is shown in Figure 1 and Figure 2.

Figure 2. Workflow of Image Grading With Quadrant-Based Diagnosis.

Figure 2.

Open in a new tab

Statistical Analysis

Two-level plus disease diagnosis (plus or not plus) was used for all statistical analyses. Intragrader agreement between eye-based and quadrant-based diagnosis and between individual quadrant (superonasal, inferonasal, superotemporal, and inferotemporal quadrant) assessment and eye-based diagnosis was analyzed using absolute agreement and κ statistic for 6 graders. Accuracy of each diagnostic approach and individual quadrant assessment compared with the RSD was also analyzed. Intergrader agreement of 6 graders in eye-based and quadrant-based diagnosis was analyzed using absolute agreement, κ statistic, and intraclass correlation coefficient. Cohen and Fleiss κ statistics were used for assessing agreement between 2 graders and among more than 2 graders, respectively. The symmetry of disagreement between eye-based and quadrant-based diagnosis was evaluated with McNemar and binomial tests, which determined whether the quadrant-based diagnosis undercalled or overcalled plus disease compared with eye-based diagnosis.

Data analysis was performed using SPSS Statistics version 24 (IBM), Microsoft Excel for Mac version 15.33 (Microsoft), R version 3.4.1 (The R Foundation), and the web-based Kappa Program.26 The κ statistic was interpreted using a commonly accepted scale: 0.21 to 0.40 indicated fair agreement, 0.41 to 0.60, moderate agreement, 0.61 to 0.80, substantial agreement, and 0.81 to 1.0, near-perfect agreement.27 All P values were 2-tailed, and significance was set at P < .05.

Results

Intragrader Reliability

eTable 1 in the Supplement shows intragrader agreement between eye-based and quadrant-based diagnosis by 6 graders. Cohen κ ranged from 0.32 to 0.75, showing variable agreement (fair to substantial) between eye-based and quadrant-based plus disease diagnoses. McNemar tests indicated statistically significant discordance for plus disease diagnosis for 5 of 6 graders between eye-based and quadrant-based methods (eTable 1 in the Supplement). Among the 5 graders with discordant diagnoses, 4 showed a discrepancy in the direction of not plus disease for quadrant-based diagnosis (eTable 1 in the Supplement).

Figure 3 and Figure 4 show a representative case of the discrepancy between eye-based and quadrant-based diagnosis. The RSD for this eye was plus disease, and all 6 graders diagnosed the eye as plus disease based on the full retinal image (Figure 3). However, the quadrant-based diagnosis was not plus disease by all 6 graders (ie, all 6 graders diagnosed fewer than 2 quadrants as plus disease when viewed individually) (Figure 4).

Figure 3. Representative Case of Eye-Based Diagnosis Showing the Entire Posterior Retinal Image.

Figure 3.

Open in a new tab

All 6 graders diagnosed this eye as plus disease based on the entire retinal image. The numerical value indicates the number of graders classifying the image as plus disease (first number), pre-plus disease (middle number), or normal (last number).

Figure 4. Representative Case of Quadrant-Based Diagnosis Showing the Cropped Quadrant Images.

Figure 4.

Open in a new tab

Cropped quadrant images of Figure 3. Quadrant-based diagnosis, which combines independent grading of individual quadrants, was not plus disease by all 6 graders. The numerical value on each image indicates the number of graders classifying that image as plus disease (first number), pre-plus disease (middle number), or normal (last number).

Agreement between individual quadrant assessment and eye-based diagnosis or RSD in all 197 images found no remarkable differences between individual quadrants compared with eye-based diagnosis or RSD (eTables 2 and 3 in the Supplement). However, when analyzing 31 images with RSD of plus disease, the inferotemporal quadrant showed higher absolute agreement with eye-based diagnosis than the inferonasal quadrant in 5 of 6 graders (eTable 4 in the Supplement), the inferotemporal quadrant showed higher absolute agreement with RSD than the inferonasal quadrant in 5 of 6 graders, and the superotemporal quadrant showed higher absolute agreement with RSD than the inferonasal quadrant in 4 of 6 graders (eTable 5 in the Supplement).

Intergrader Reliability

Intergrader agreement among the 6 graders is shown in the Table. Cohen κ statistic of pair-wise agreement between all graders indicated that eye-based diagnosis showed substantial to near-perfect agreement. By contrast, quadrant-based diagnosis showed only fair to substantial agreement. Fleiss multigrader κ statistic showed substantial agreement for eye-based diagnosis while quadrant-based diagnosis had only moderate agreement. Moreover, nonoverlapping 95% confidence intervals of Fleiss κs and intraclass correlation coefficients suggest that intergrader agreement for eye-based diagnosis was higher than for quadrant-based diagnosis and individual quadrant diagnosis.

Table. Intergrader Reliability for Plus Disease Diagnosis Using Eye-Based and Quadrant-Based Approachesa.

Statistic Eye-Based Diagnosis (n = 197 Eyes) Quadrant-Based Diagnosis (n = 197 Eyes) Single Quadrant Diagnosis (n = 788 Quadrants)
Absolute agreement, mean (SD; range) 0.93 (0.01; 0.89-0.96) 0.90 (0.03; 0.82-0.97) 0.92 (0.01; 0.84-0.97)
Cohen κ, mean (SD; range) 0.75 (0.03; 0.62-0.85) 0.56 (0.07; 0.31-0.74) 0.49 (0.05; 0.30-0.67)
Fleiss κ (95% CI) 0.75 (0.71-0.78)b 0.55 (0.51-0.59)b 0.49 (0.47-0.51)b
Intraclass correlation coefficient (95% CI) 0.95 (0.94-0.96)b 0.89 (0.87-0.92)b 0.87 (0.85-0.88)b
Open in a new tab
a

Six expert graders evaluated 197 posterior retinal images (31 with reference standard diagnosis of plus disease and 166 with reference standard of pre-plus disease or normal).

b

Nonoverlapping 95% CIs suggest that intergrader agreement for eye-based diagnosis was higher than those of quadrant-based diagnosis and single quadrant diagnosis.

Diagnostic Accuracy

Cohen κ statistic showed that the accuracy of eye-based diagnosis by the 6 graders compared with the RSD was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader (eTable 6 in the Supplement). The 95% confidence intervals of 2 graders do not overlap between eye-based and quadrant-based plus diagnosis (eTable 6 in the Supplement).

Discussion

This study assessed whether grading plus disease individually by quadrant provided the same diagnosis as grading at the whole eye level. There are 3 key findings. First, intragrader agreement between eye-based and quadrant-based diagnosis is limited. There was variability among the graders, and 4 of 6 graders underdiagnosed plus disease when using a quadrant-based approach compared with an eye-based approach. Second, intergrader agreement with quadrant-based diagnosis was lower than with eye-based diagnosis. Third, diagnostic accuracy with quadrant-based diagnosis was lower than with eye-based diagnosis. Taken together, these findings suggest that eye-based diagnosis may have advantages over quadrant-based diagnosis regarding reliability and accuracy.

The first key finding is that intragrader agreement between eye-based and quadrant-based plus diagnosis is imperfect. There are several potential explanations. First, real-world assessment of plus disease grading in individual quadrants may be influenced by the appearance of adjacent quadrants. This could explain the underdiagnosis of plus disease in this study when the masked quadrant-based approach was applied (eg, Figure 3 and Figure 4). In other words, although this study did not formally address this question, it may be that the grading of individual quadrants using telemedicine or ophthalmoscopy is influenced by the presence of information in adjacent quadrants and yields a different quadrant-based diagnosis, thus potentially yielding a different overall diagnosis even for examiners trying to apply a strict quadrant-level approach based on the definition. A 2017 study28 also suggested that that even experts could be biased by factors other than retinal image findings (eg, gestational age). Because both ophthalmoscopic and image-based diagnosis provides information about all quadrants simultaneously, interaction between quadrants may be inevitable in a real clinical setting, and eye-based diagnosis is presumably a more natural method for retinal assessment. Second, when assessing plus disease, graders may have different interpretations regarding the criterion of 2 or more quadrants of arterial dilation and venous tortuosity as shown in the standard published photograph, particularly given that the photograph has differing findings in each quadrant.5,6

The second key finding is that quadrant-based diagnosis showed lower intergrader agreement than eye-based diagnosis. In this study, the agreement of quadrant-based diagnosis between graders was lower than that of eye-based diagnosis. Interobserver disagreement in plus disease has been well documented, including ways in which experts deviate from the published definition of plus disease, such as by incorporating features outside of the posterior pole and nonstandard features, such as venous tortuosity.9,10,11,12,13,14,15,16,17,18,19,20 Variable adherence to the number of involved quadrants between examiners could be another explanation. Additional specification regarding this definition of plus disease may improve diagnostic consistency in the future.

The third key finding is that accuracy of quadrant-based diagnosis, based on agreement with the RSD, was lower than of eye-based diagnosis. Because ophthalmologists are trained and accustomed to assess retinal information in all quadrants during standard clinical examination, it may not be surprising that experts performed better when given the information from the whole rather than an individual quadrant. Nonetheless, it is important to be aware that the 2 approaches (eye-based vs quadrant-based) do not always give the same diagnosis. Based on these data, it is hard to argue for a strict interpretation of a quadrant-based approach because it is less likely to produce a diagnosis that agrees with the majority of expert observers.

To measure accuracy in this study, we calculated agreement of each grader’s diagnosis with an RSD, as described above.25 We have shown in previous studies17 that an RSD for plus disease developed using these methods agreed with the majority vote of clinical experts in nearly all cases, suggesting very high external validity of our RSD. In other studies,15 we have also demonstrated that the average plus disease severity score from clinical experts was very close to this RSD. Therefore, we believe that agreement with this RSD may be interpreted as diagnostic accuracy, with higher validity than ophthalmoscopic or image-based diagnosis by an individual expert.

There is growing clinical evidence that interobserver disagreement in plus disease diagnosis yields clinically meaningful differences in patient care. In the Cryotherapy for ROP study,5 examiners disagreed on 12% of cases with threshold ROP.10 Since that time, to our knowledge, there has been very little direct comparison of treatment decisions between experts outside of research studies that have consistently shown imperfect agreement.9,11,12,13,14,15,16,17,18 However, 2018 results from the Benefits of Oxygen Saturation Targeting (BOOST)-II trial29 found evidence of variation in treatment recommendations between physicians for the same level of disease based on fundus photographs. This reaffirms the importance of addressing the underlying causes and finding solutions for the wide variation in plus disease diagnosis. Despite the accepted definition of plus disease, there is now evidence that clinicians incorporate information from outside the posterior pole,17 assign diagnostic importance to features that are not in the official definition (eg, venous tortuosity),17 disagree on the level of dilation and tortuosity sufficient for diagnosis of plus disease,15 and may use eye-based or quadrant-based approaches.11,17,21,22,23 In other words, there is not a single part of the official definition of plus disease that consistently corresponds to expert behavior, which suggests that a more explicit definition may be useful.

An increasingly popular approach toward making plus disease diagnosis more objective and quantitative is computer-based image analysis.30,31,32,33,34 To date, development of these systems has been deductive, starting with the components of the accepted definition of plus disease and evaluating various defined vascular features within this context. For example, one system (ROPTool) uses metrics of arterial tortuosity and venous dilation and combines the values from the 2 most extreme quadrants to detect plus disease.30,35 Findings from the current study suggest that deductive computer-based algorithms that are eye-based rather than quadrant-based may be more successful.16,17 Several computer-based image analysis systems have been developed that use various combinations of quantitative measures to objectively measure plus disease severity.30,31,32 In addition to their not being widely available and requiring varying amounts of user input to produce a diagnosis, widespread use of these systems has been limited by their variable agreement with clinical plus disease diagnosis.17,22,30,34 Moreover, results from the current study add to the prior literature demonstrating how the clinical diagnosis of plus disease often deviates from the published definition. Therefore, an inductive approach to algorithm development, such as with deep learning, may better produce results that correspond to clinical diagnosis.36 By analyzing the outputs of deep learning–based algorithms, we might gain insights into the inductive diagnostic process used by experts.

More broadly, these study findings have implications for diagnosis of other retinal diseases and for emerging diagnostic modalities, such as telemedicine. In diseases such as diabetic retinopathy, large-scale clinical trials have developed methods for disease classification that are quadrant-based (eg, the 4-2-1 rule for the diagnosis of severe nonproliferative diabetic retinopathy37). Similarly, reading centers have been developed for remote telemedicine grading of diseases, such as diabetic retinopathy and ROP, which may be operated by trained nonphysician readers who interpret images based on specific algorithms.38,39,40 In these other settings, analysis of agreement between eye-based vs quadrant-based diagnosis may be warranted.

Limitations

There are several limitations of this study. First, only 6 graders were included. This may limit generalizability of our findings. However, we note that this topic has never been studied before, to our knowledge. Second, there are technical limitations on image grading. Most study images included more visible temporal than nasal retina, and the areas displayed for each quadrant were not consistent among different images, which may have affected both eye-based and image-based diagnosis, as previous studies have shown that field of view could affect the plus disease diagnosis.17,20 Standardizing areas of each quadrant by generating multi-image mosaics may overcome this limitation in future studies. In addition, dividing images into quadrants was done by vertical and horizontal lines passing through the center of the optic disc, which may not have completely matched with the anatomical quadrants. This might have limited grader’s ability to access the entire contour of the vasculature, especially when quadrant cutoff divided major vessels. Third, 3 of the 6 graders contributed to establishing the RSD for the analyzed images, which may have created bias. However, the RSD was determined by combining the image-based diagnoses of 3 graders with the actual clinical diagnosis by an independent expert, and this study was performed several years after the initial RSD was established for each image. Thus, we feel it is unlikely that the results of this study were significantly affected by overlap in graders. Fourth, this study did not assess the importance of pre-plus disease in the diagnosis because ICROP provides no guidance on the relevance of the numbers of quadrants of pre-plus disease involved.

Conclusions

We believe these study findings provide guidance for ophthalmologists about the best way to diagnose plus disease and that this has important implications for quality of care, delivery of care, and education. In the future, we feel that more precise definitions of plus disease regarding the criterion requiring 2 or more quadrants of abnormality will provide additional education and diagnostic standardization for practicing ophthalmologists.41,42,43 Next-generation computer-based image analysis systems as well as imaging modalities such as optical coherence tomography and optical coherence tomographic angiography may build on this work toward developing more objective and reproducible diagnostic methods. These future metrics will require prospective validation to determine when the appropriate intervention to treat ROP should be, but our hope is that in developing these new definitions, we can at least all agree.

Supplement.

eTable 1. Intragrader reliability between eye-based and quadrant-based diagnosis.

eTable 2. Intragrader agreement between individual quadrant assessment and eye-based approach for plus disease diagnosis.

eTable 3. Agreement between individual quadrant assessment and reference standard diagnosis.

eTable 4. Intragrader agreement between individual quadrant assessment and eye-based approach for plus disease diagnosis in 31 images with reference standard diagnosis of plus disease.

eTable 5. Agreement between individual quadrant assessment and reference standard diagnosis in 31 images with reference standard diagnosis of plus disease.

eTable 6. Diagnostic accuracy of eye-based vs quadrant-based plus disease diagnosis compared with a consensus reference standard diagnosis.

Click here for additional data file. (55.2KB, pdf)
Journal Club
Click here for additional data file. (984.6KB, pptx)

References

  • 1.Gilbert C. Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84(2):77-82. [DOI] [PubMed] [Google Scholar]
  • 2.Hartnett ME. Pathophysiology and mechanisms of severe retinopathy of prematurity. Ophthalmology. 2015;122(1):200-210. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.International Committee for the Classification of Retinopathy of Prematurity An international classification of retinopathy of prematurity. Pediatrics. 1984;74(1):127-133. [PubMed] [Google Scholar]
  • 4.International Committee for the Classification of Retinopathy of Prematurity The International Classification of Retinopathy of Prematurity revisited. Arch Ophthalmol. 2005;123(7):991-999. [DOI] [PubMed] [Google Scholar]
  • 5.Cryotherapy for Retinopathy of Prematurity Cooperative Group Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results. Arch Ophthalmol. 1988;106(4):471-479. [DOI] [PubMed] [Google Scholar]
  • 6.Early Treatment For Retinopathy Of Prematurity Cooperative Group Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121(12):1684-1694. [DOI] [PubMed] [Google Scholar]
  • 7.The STOP-ROP Multicenter Study Group Supplemental Therapeutic Oxygen for Prethreshold Retinopathy Of Prematurity (STOP-ROP), a randomized, controlled trial: I: primary outcomes. Pediatrics. 2000;105(2):295-310. [DOI] [PubMed] [Google Scholar]
  • 8.Mintz-Hittner HA, Kennedy KA, Chuang AZ; BEAT-ROP Cooperative Group . Efficacy of intravitreal bevacizumab for stage 3+ retinopathy of prematurity. N Engl J Med. 2011;364(7):603-615. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875-880. [DOI] [PubMed] [Google Scholar]
  • 10.Reynolds JD, Dobson V, Quinn GE, et al. ; CRYO-ROP and LIGHT-ROP Cooperative Study Groups . Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120(11):1470-1476. [DOI] [PubMed] [Google Scholar]
  • 11.Wallace DK, Quinn GE, Freedman SF, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS. 2008;12(4):352-356. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Gschließer A, Stifter E, Neumayer T, et al. Inter-expert and intra-expert agreement on the diagnosis and treatment of retinopathy of prematurity. Am J Ophthalmol. 2015;160(3):553-560.e3. [DOI] [PubMed] [Google Scholar]
  • 13.Slidsborg C, Forman JL, Fielder AR, et al. Experts do not agree when to treat retinopathy of prematurity based on plus disease. Br J Ophthalmol. 2012;96(4):549-553. [DOI] [PubMed] [Google Scholar]
  • 14.Hewing NJ, Kaufman DR, Chan RV, Chiang MF. Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts. JAMA Ophthalmol. 2013;131(8):1026-1032. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Campbell JP, Kalpathy-Cramer J, Erdogmus D, et al. ; Imaging and Informatics in Retinopathy of Prematurity Research Consortium . Plus disease in retinopathy of prematurity: a continuous spectrum of vascular abnormality as a basis of diagnostic variability. Ophthalmology. 2016;123(11):2338-2344. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Kalpathy-Cramer J, Campbell JP, Erdogmus D, et al. ; Imaging and Informatics in Retinopathy of Prematurity Research Consortium . Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis. Ophthalmology. 2016;123(11):2345-2351. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Campbell JP, Ataer-Cansizoglu E, Bolon-Canedo V, et al. ; Imaging and Informatics in ROP (i-ROP) Research Consortium . Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis. JAMA Ophthalmol. 2016;134(6):651-657. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Campbell JP, Ryan MC, Lore E, et al. ; Imaging & Informatics in Retinopathy of Prematurity Research Consortium . Diagnostic discrepancies in retinopathy of prematurity classification. Ophthalmology. 2016;123(8):1795-1801. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Keck KM, Kalpathy-Cramer J, Ataer-Cansizoglu E, You S, Erdogmus D, Chiang MF. Plus disease diagnosis in retinopathy of prematurity: vascular tortuosity as a function of distance from optic disk. Retina. 2013;33(8):1700-1707. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Rao R, Jonsson NJ, Ventura C, et al. Plus disease in retinopathy of prematurity: diagnostic impact of field of view. Retina. 2012;32(6):1148-1155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Wallace DK, Freedman SF, Zhao Z, Jung SH. Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity. Arch Ophthalmol. 2007;125(11):1523-1530. [DOI] [PubMed] [Google Scholar]
  • 22.Abbey AM, Besirli CG, Musch DC, et al. Evaluation of screening for retinopathy of prematurity by ROPtool or a lay reader. Ophthalmology. 2016;123(2):385-390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Daniel E, Quinn GE, Hildebrand PL, et al. ; e-ROP Cooperative Group . Validated system for centralized grading of retinopathy of prematurity: telemedicine approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. JAMA Ophthalmol. 2015;133(6):675-682. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.World Medical Association World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191-2194. doi: 10.1001/jama.2013.281053. [DOI] [PubMed] [Google Scholar]
  • 25.Ryan MC, Ostmo S, Jonas K, et al. Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology. AMIA Annu Symp Proc. 2014;2014:1902-1910. [PMC free article] [PubMed] [Google Scholar]
  • 26.StatsToDo Kappa (Cohen and Fleiss) for ordinal data program. https://www.statstodo.com/CohenFleissKappa_Pgm.php. Accessed September 5, 2017.
  • 27.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174. [PubMed] [Google Scholar]
  • 28.Gschließer A, Stifter E, Neumayer T, et al. Effect of patients’ clinical information on the diagnosis of and decision to treat retinopathy of prematurity [published online October 3, 2017]. Retina. 2017. doi: 10.1097/IAE.0000000000001864 [DOI] [PubMed] [Google Scholar]
  • 29.Fleck BW, Williams C, Juszczak E, et al. ; BOOST II Retinal Image Digital Analysis (RIDA) Group . An international comparison of retinopathy of prematurity grading performance within the Benefits of Oxygen Saturation Targeting II trials. Eye (Lond). 2018;32(1):74-80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Wittenberg LA, Jonsson NJ, Chan RV, Chiang MF. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity. J Pediatr Ophthalmol Strabismus. 2012;49(1):11-19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Wallace DK. Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc. 2007;105:594-615. [PMC free article] [PubMed] [Google Scholar]
  • 32.Koreen S, Gelman R, Martinez-Perez ME, et al. Evaluation of a computer-based system for plus disease diagnosis in retinopathy of prematurity. Ophthalmology. 2007;114(12):e59-e67. [DOI] [PubMed] [Google Scholar]
  • 33.Gelman R, Martinez-Perez ME, Vanderveen DK, Moskowitz A, Fulton AB. Diagnosis of plus disease in retinopathy of prematurity using Retinal Image Multiscale Analysis. Invest Ophthalmol Vis Sci. 2005;46(12):4734-4738. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Ataer-Cansizoglu E, Bolon-Canedo V, Campbell JP, et al. ; i-ROP Research Consortium . Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis. Transl Vis Sci Technol. 2015;4(6):5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Cabrera MT, Freedman SF, Kiely AE, Chiang MF, Wallace DK. Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity. J AAPOS. 2011;15(1):40-44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Kalpathy-Cramer J, Campbell JP, Kim S, et al. Deep learning for the identification of plus disease in retinopathy of prematurity [ARVO abstract 5554]. Invest Ophthalmol Vis Sci. 2017;58:5554. [Google Scholar]
  • 37.Wilkinson CP, Ferris FL III, Klein RE, et al. ; Global Diabetic Retinopathy Project Group . Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003;110(9):1677-1682. [DOI] [PubMed] [Google Scholar]
  • 38.Quinn GE, Ying GS, Daniel E, et al. ; e-ROP Cooperative Group . Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity. JAMA Ophthalmol. 2014;132(10):1178-1184. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Daskivich LP, Vasquez C, Martinez C Jr, Tseng CH, Mangione CM. Implementation and evaluation of a large-scale teleretinal diabetic retinopathy screening program in the Los Angeles County Department of Health Services. JAMA Intern Med. 2017;177(5):642-649. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Vinekar A, Gilbert C, Dogra M, et al. The KIDROP model of combining strategies for providing retinopathy of prematurity screening in underserved areas in India using wide-field imaging, tele-medicine, non-physician graders and smart phone reporting. Indian J Ophthalmol. 2014;62(1):41-49. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Myung JS, Paul Chan RV, Espiritu MJ, et al. Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training. J AAPOS. 2011;15(6):573-578. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Patel SN, Martinez-Castellanos MA, Berrones-Medina D, et al. ; GEN-ROP; i-ROP Research Consortium . Assessment of a tele-education system to enhance retinopathy of prematurity training by international ophthalmologists-in-training in Mexico. Ophthalmology. 2017;124(7):953-961. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Nagiel A, Espiritu MJ, Wong RK, et al. Retinopathy of prematurity residency training. Ophthalmology. 2012;119(12):2644-5.e1, 2. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement.

eTable 1. Intragrader reliability between eye-based and quadrant-based diagnosis.

eTable 2. Intragrader agreement between individual quadrant assessment and eye-based approach for plus disease diagnosis.

eTable 3. Agreement between individual quadrant assessment and reference standard diagnosis.

eTable 4. Intragrader agreement between individual quadrant assessment and eye-based approach for plus disease diagnosis in 31 images with reference standard diagnosis of plus disease.

eTable 5. Agreement between individual quadrant assessment and reference standard diagnosis in 31 images with reference standard diagnosis of plus disease.

eTable 6. Diagnostic accuracy of eye-based vs quadrant-based plus disease diagnosis compared with a consensus reference standard diagnosis.

Click here for additional data file. (55.2KB, pdf)
Journal Club
Click here for additional data file. (984.6KB, pptx)

Articles from JAMA Ophthalmology are provided here courtesy of American Medical Association

RESOURCES