Detection of Potentially Severe Retinopathy of Prematurity by Remote Image Grading

Abstract

IMPORTANCE

Telemedicine in retinopathy of prematurity (ROP) has the potential for delivering timely care to premature infants at risk for serious ROP.

OBJECTIVE

To describe the characteristics of eyes at risk for ROP to provide insights into what types of ROP are most easily detected early by image grading.

DESIGN, SETTING, AND PARTICIPANTS

Secondary analysis of eyes with referral-warranted (RW) ROP (stage 3 ROP, zone I ROP, plus disease) on diagnostic examination from the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) study was conducted from May 1, 2011, to October 31, 2013, in 1257 premature infants with birth weights less than 1251 g in 13 neonatal units in North America. Data analysis was performed between February 1, 2016, and June 5, 2017.

INTERVENTIONS

Serial imaging sessions with concurrent diagnostic examinations for ROP.

MAIN OUTCOMES AND MEASURES

Time of detecting RW-ROP on image evaluation compared with clinical examination.

RESULTS

In the e-ROP study, 246 infants (492 eyes) were included in the analysis; 138 (56.1%) were male. A total of 447 eyes had RW-ROP on diagnostic examination. Image grading in 123 infants (mean [SD] gestational age, 24.8 [1.4] weeks) detected RW-ROP earlier than diagnostic examination (early) in 191 (42.7%) eyes by about 15 days and detected RW-ROP in 123 infants (mean [SD] gestational age, 24.6 [1.5] weeks) at the same time (same) in 200 (44.7%) eyes. Most of the early eyes (153 [80.1%]) interpreted as being RW-ROP positive on imaging evaluation agreed with examination findings when the examination subsequently documented RW-ROP. At the sessions in which RW-ROP was first found by examination, stage 3 or more in 123 infants (mean [SD] gestational age, 24.8 [1.4] weeks) ROP was noted earlier on image evaluation in 151 of 191 early eyes (79.1%) and in 172 of 200 of same eyes (86.0%) (P = .08); the presence of zone I ROP was detected in 57 of 191 (29.8%) early eyes vs 64 of 200 (32.0%) same eyes (P = .90); and plus disease was noted in 30 of 191 (15.7%) early eyes and 45 of 200 (22.5%) same eyes (P = .08).

CONCLUSIONS AND RELEVANCE

In both early and same eyes, zone I and/or stage 3 ROP determined a significant proportion of RW-ROP; plus disease played a relatively minor role. In most early RW-ROP eyes, the findings were consistent with clinical examination and/or image grading at the next session. Because ROP telemedicine is used more widely, development of standard approaches and protocols is essential.

Recent studies evaluating the validity of telemedicine approaches to retinopathy of prematurity (ROP) have high-lighted strengths and weaknesses of remote evaluation of digital images from at-risk infants compared with clinical examination by an ophthalmologist.^1–4 In the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) study, digital images obtained from 1257 at-risk infants with birth weight (BW) less than 1251 g were graded remotely by trained nonphysician readers, using a structured study protocol.^5,6 The e-ROP study found high sensitivity (90.0%) and specificity (87.0%) for detecting infants who warranted a referral to an ophthalmologist to consider treatment; for example, 1 or both eyes had referral-warranted (RW) ROP, defined as zone I ROP, stage 3 ROP or more, or plus disease. In a recent report,⁷ e-ROP investigators presented a review by a consensus panel of ROP experts that examined 161 false-negative and 854 false-positive images from eyes in which findings consistent with RW-ROP were noted either on diagnostic examination or grading, but not both. Based on the panel results, it was estimated that the ROP experts would agree with the clinical examination findings in 46.5% (95% CI, 41.6%–51.6%) of the false-negative cases (ie, RW-ROP found by examination but not by image evaluation) and agree with trained readers’ grading in 70.0% (95% CI, 67.3%–72.8%) of the false-positive cases.

The following question is addressed in the present report. Among 447 eyes with RW-ROP on diagnostic examination, what are the characteristics of eyes for which image grading detected RW-ROP earlier than diagnostic examination (early) compared with eyes with RW-ROP detected at the same time as the examination (same)?

Methods

This is a secondary analysis of e-ROP data, an observational cohort study of 1257 premature infants conducted in 13 North American centers from May 1, 2011, to October 31, 2013. The e-ROP study protocol and informed consent process were approved by institutional review boards of participating centers and this secondary analysis was approved by the e-ROP Executive Committee. Infants underwent scheduled diagnostic examinations and digital imaging. Trained nonphysician readers graded images for the presence of RW-ROP. Details of results and procedures have been previously published.^6,8–13

Statistical Analysis

We compared early eyes with same eyes for BW, gestational age, and postmenstrual age at the image session and characteristics of RW-ROP components. Means were compared using a 2-sided, unpaired t test, and Fisher exact test was used for proportions. Data analysis was performed between February 1, 2016, and June 5, 2017. P < .05 was considered statistically significant. Statistical analysis was conducted using SAS, version 9.4 (SAS Institute Inc).

Results

In the e-ROP study, 246 infants (492 eyes) were included in the analysis; 138 (56.1%) were male. Among the infants (123 early, 123 same), mean (SD) BW was similar (early, 699 [172] vs same, 667 [149] g; P = .10), as was the mean gestational age (24.8 [1.4] vs 24.6 [1.5] weeks; P = .28). RW-ROP was detected on image grading at a mean postmenstrual age of 34 weeks for early eyes and 36 weeks for same eyes (P = .01). By 34 weeks’ postmenstrual age, grading detected RW-ROP in 114 (59.7%) early eyes compared with 73 (36.5%) same eyes (P < .001). Image grading noted RW-ROP in 76 (40.0%) of the 191 early eyes within 1 week of a subsequent clinical RW-ROP diagnosis, and 9% of the early eyes were diagnosed more than 4 weeks later. In the Figure, a series of retinal images from an e-ROP infant is presented in which image evaluation documented stage 3 ROP from approximately 3 weeks before it was diagnosed on clinical examination and confirmed again on subsequent examination a week later.

Figure. Series of Retinal Images Over 22 Days From the Left Eye of an Infant in the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (ROP) Study.

A, Stage 3 ROP by image reader and as stage 1 ROP on clinical examination. B, Stage 3 ROP by image reader and as stage 2 ROP on clinical examination 8 days later. C, Stage 3 ROP by image reader and clinical examination 15 days later. D, Stage 3 ROP by image reader and clinical examination 22 days later.

The ROP status on clinical examination when RW-ROP was first noted was significantly different for early vs same eyes for zone of ROP and plus disease (P < .001), but not for ROP stage (P = .39) (Table 1). When a single RW-ROP component was present, stage 3 was significantly more common in early eyes than same eyes (35 [18.3%] vs 13 [6.5%], P < .001). When 2 or 3 components of RW-ROP were considered, same eyes had more components noted than early eyes (72 [36.0%] vs 48 [25.1%], P < .001). In 16 (8.4%) early eyes and 9 (4.5%) same eyes, the presence of RW-ROP could not be determined on image grading.

Table 1.

ROP Status When RW-ROP Was First Detected on Clinical Examination

Characteristic	RW-ROP Detection, No. (%)		P Value
	Imaging Earlier Than Examination (n = 191 eyes [123 infants])	Same Session for Imaging and Examination (n = 200 eyes [123 infants])
Clinical Examination
ROP stage
No ROP	0	0
1	4 (2.1)	9 (4.5)	.39
2	24 (12.6)	24 (12.0)
≥3	163 (85.3)	165 (82.5)
Cannot grade	0	2 (1.0)
Zone of ROP
No ROP	0	2 (1.0)	<.001
1	13 (6.8)	47 (23.5)
2	178 (93.2)	151 (75.5)
Posterior pole
Normal	55 (28.8)	41 (20.5)	<.001
Preplus disease	78 (40.8)	58 (29.0)
Plus disease	58 (30.4)	101 (50.5)
Cannot grade	0	0
Image Grading
ROP stage
No ROP	4 (2.1)	2 (1.0)	.08
1	0	0
2	30 (15.7)	17 (8.5)
≥3	151 (79.1)	172 (86.0)
Cannot grade	6 (3.1)	9 (5)
Zone of ROP
No ROP	9 (4.7)	9 (4.5)	.90
1	57 (29.8)	64 (32.0)
2	125 (65.4)	127 (63.5)
Posterior pole
Normal	39 (20.4)	27 (13.5)	.08
Preplus disease	116 (60.7)	120 (60.0)
Plus disease	30 (15.7)	45 (22.5)
Cannot grade	6 (3.1)	8 (4.0)

Abbreviations: ROP, retinopathy of prematurity; RW, referral-warranted.

At the first RW-ROP detection by image evaluation, stage 3 ROP (eTable 1 in the Supplement) was noted on images in 138 (72.3%) early eyes and 152 (76.0%) same eyes. More early eyes were graded as having zone I ROP (79 [41.4%]) than same eyes (64 [32.0%]), while only 42 (21.0%) same eyes had zone I ROP based on examination (eTable 2 in the Supplement). In 15 (7.9%) early eyes and 7 (3.5%) same eyes, the zone or presence of ROP could not be determined. Among early eyes, plus disease (eTable 3 in the Supplement) was noted in 9 (4.7%) eyes and preplus disease was detected in 95 (49.7%) eyes, but the examination noted preplus disease in only 48 eyes (25.1%). Among same eyes, 99 (49.5%) had plus disease on examination and 45 (22.5%) eyes were graded as having plus disease on imaging. In 11 (5.8%) early eyes and 7 (3.5%) same eyes, the presence of plus disease could not be determined on image grading.

Among the 191 early eyes, the grading of image sets obtained subsequently found that 152 (79.6%) eyes were again graded as RW-ROP at the next session and also in 153 (80.1%) at the last image session. When RW-ROP was first detected by clinical examination, image evaluations agreed with examination in 166 (86.9%) of the eyes. Among early eyes, RW-ROP findings were noted on clinical examination in 90 (47.1%) eyes at the next session and in 157 (82.2%) eyes at the last image set obtained (Table 2).

Table 2.

Image Grading After First Detection of RW-ROP Among the 191 Eyes With RW-ROP Detected Earlier in Image Grading Than Clinical Examination

RW-ROP Status	No. (%)
	First Subsequent Image After First Detection on Imaging	Last Image After First Detection on Imaging	Image Session After First Detection on Clinical Examination
Image grading
Absent in image grading	39 (20.4)	38 (19.9)	25 (13.1)
Present in image grading	152 (79.6)	153 (80.1)	166 (86.9)
Clinical examination
Absent on clinical examination	101 (52.9)	34 (17.8)	0
Present on clinical examination	90 (47.1)	157 (82.2)	191 (100)

Abbreviations: ROP, retinopathy of prematurity; RW, referral-warranted.

Discussion

In this study, we examined 2 groups of RW-ROP eyes based on clinical examination from the e-ROP study: early eyes (RW-ROP detected on image grading before clinical examination) and same eyes (RW-ROP detected at the same session). In most early and same eyes, zone I ROP was a factor that determined the designation for RW-ROP and stage 3 ROP accounted for a significant proportion; however, plus disease played a relatively minor role in the RW-ROP determination. The early image grading results were consistent with the RW-ROP diagnosis on clinical examination at subsequent sessions (79.6% (152 of 191) at the next session and 80.1% (153 of 191) at the final session).

There is a growing consensus of the utility of remote evaluation of images from infants at ROP risk, and steps toward validation of such a program need to be systematically developed. The shift away from clinical examination to using digital images to determine the need for an ROP examination has several advantages: the extension of ROP surveillance to regions in which examination by experienced ophthalmologists is limited; standard grading protocols; and improved imaging systems will likely enable a more reproducible and reliable provision of care to at-risk infants.

Limitations

One important consideration when comparing the imaging grading in ROP with the results of the clinical examination is that there are likely to be discrepancies between the 2 approaches, some of which can be explained by errors in clinical examination or in the image grading. One possible explanation for such discrepancies is that static observation of remotely acquired digital images and in-person examination of an infant in an intensive care unit may provide different opportunities for detecting severe ROP. In the e-ROP study, we estimated agreement with the examination in 46.5% at the next session and 70.0% at the final session of the false-positive e-ROP cases and agreement with image grading in 70.0% of the false-negative cases.⁷ This finding suggests that development of standard approaches, especially as imaging technology improves, is essential for optimum care to infants at ROP risk.

Conclusions

This report supplements our understanding of telemedicine in ROP. We found that early detection of RW-ROP on image grading is likely to be early detection of severe disease and not necessarily an error in grading. There is a need to develop standard approaches to the use of digital images as this technology becomes more widely dispersed in the United States as well as other countries. It is essential that a stepwise investigation similar to that done in telehealth in patients with diabetes be undertaken.^14,15

Key Points.

Question

What types of potentially serious retinopathy of prematurity are most likely to be detected earlier on image grading than on clinical examination?

Findings

Secondary analysis of data from the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity cohort study was conducted in 191 infants to determine when image evaluation and clinical examination detected potentially serious retinopathy of prematurity. In most eyes in which serious retinopathy of prematurity was noted on image grading before it was documented on clinical examination, the image finding was consistent with the examination and/or image grading at the next session.

Meaning

With increasing use of retinopathy of prematurity telemedicine in the United States and worldwide, developing standard approaches for image grading are critical.

Appendix

Group Information: The Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity study investigators included the following: Office of Study Chair: The Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania: Graham E. Quinn, MD, MSCE (principal investigator [PI]); Kelly Wade, MD, PhD, MSCE; Agnieshka Baumritter, MS; Trang B. Duros, BA; and Lisa Erbring. Study centers: Johns Hopkins University, Baltimore, Maryland: Michael X. Repka, MD (PI); Jennifer A. Shepard, CRNP; Pamela Donohue, ScD; David Emmert, BA; and C. Mark Herring, CRA. Boston Children’s Hospital, Boston, Massachusetts: Deborah VanderVeen, MD; Suzanne Johnston, MD; Carolyn Wu, MD; Jason Mantagos, MD; Danielle Ledoux, MD; Tamar Winter, RN, BSN, IBCLC; Frank Weng, BS; and Theresa Mansfield, RN, BSN. Nationwide Children’s Hospital, Columbus, Ohio, and Ohio State University Hospital, Columbus: Don L. Bremer, MD (PI); Richard Golden, MD; Mary Lou McGregor, MD; Catherine Olson Jordan, MD; David Rogers, MD; Rae R. Fellows, MEd, CCRC; Suzanne Brandt, RNC, BSN; and Brenda Mann, RNC, BSN. Duke University, Durham, North Carolina: David Wallace, MD (PI); Sharon Freedman, MD; Sarah K. Jones, BS; Du Tran-Viet, BS; and Rhonda “Michelle” Young, RN. University of Louisville, Louisville, Kentucky: Charles C. Barr, MD (PI); Rahul Bhola, MD; Craig Douglas, MD; Peggy Fishman, MD; Michelle Bottorff, BS; Brandi Hubbuch, RN, MSN, NNP-BC; and Rachel Keith, PhD. University of Minnesota, Minneapolis: Erick D. Bothun, MD (PI); Inge DeBecker, MD; Jill Anderson, MD; Ann Marie Holleschau, BA, CCRP; Nichole E. Miller, MA, RN, NNP; and Darla N. Nyquist, MA, RN, NNP. University of Oklahoma, Oklahoma City: R. Michael Siatkowski, MD (PI); Lucas Trigler, MD; Marilyn Escobedo, MD; Karen Corff, MS, ARNP, NNP-BC; Michelle Huynh-Blunt, MS, ARNP; and Kelli Satnes, MS, ARNP, NNP-BC; The Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania: Monte D. Mills, MD; Will Anninger, MD; Gil Binenbaum, MD, MSCE; Graham E. Quinn, MD, MSCE; Karen A. Karp, BSN; and Denise Pearson, COMT. University of Texas Health Science Center at San Antonio: Alice Gong, MD (PI); John Stokes, MD; Clio Armitage Harper, MD; Laurie Weaver, RNC, BSN; Carmen McHenry, BSN; Kathryn Conner, RN, BSN; Rosalind Heemer, BSN; Elnora Cokley, RNC; and Robin Tragus, MSN, RN, CCRC. University of Utah, Salt Lake City: Robert Hoffman, MD (PI); David Dries, MD; Katie Jo Farnsworth, BS; Deborah Harrison, MS; Bonnie Carlstrom, COA; and Cyrie Ann Frye, CRA, OCT-C. Vanderbilt University, Nashville, Tennessee: David Morrison, MD (PI); Sean Donahue, MD; Nancy Benegas, MD; Sandy Owings, COA, CCRP; Sandra Phillips, COT, CRI; and Scott Ruark, DO. Foothills Medical Center, Calgary, Alberta, Canada: Anna Ells, MD, FRCS (PI); Patrick Mitchell, MD; April Ingram, BS; and Rosie Sorbie, RN. Data Coordinating Center: Perelman School of Medicine, University of Pennsylvania, Philadelphia: Gui-shuang Ying, PhD (PI); Maureen Maguire, PhD; Mary Brightwell-Arnold, BA, SCP; Max Pistilli, MS; Kathleen McWilliams, CCRP; Sandra Harkins; and Claressa Whearry. Image Reading Center: Perelman School of Medicine, University of Pennsylvania: Ebenezer Daniel, MBBS, MS, MPH (PI); E. Revell Martin, BA; Candace R. Parker Ostroff, BA; Krista Sepielli, BFA; and Eli Smith, BA. Expert Readers: The Vision Research ROPARD Foundation, Novi, Michigan: Antonio Capone, MD. Emory University School of Medicine, Atlanta, Georgia: G. Baker Hubbard, MD. Foothills Medical Center, Calgary, Alberta, Canada: Anna Ells, MD, FRCS. Image Data Management Center: Inoveon Corporation, Oklahoma City, Oklahoma: P. Lloyd Hildebrand, MD (PI); Kerry Davis, BA; G. Carl Gibson, BBA, CPA; and Regina Hansen, COT. Cost-Effectiveness Component: Duke University, Durham, North Carolina: Alex R. Kemper, MD, MPH, MS (PI). University of Michigan, Ann Arbor: Lisa Prosser, PhD. Data Management and Oversight Committee: Kellogg Eye Center, University of Michigan, Ann Arbor: David C. Musch, PhD, MPH (chair); Department of Ophthalmology, Boston Medical Center, Boston University School of Medicine, Boston, Massachusetts: Stephen P. Christiansen, MD; Bascom Palmer Eye Institute, University of Miami Leonard M. Miller School of Medicine, Miami, Florida: Ditte J. Hess, CRA; Department of Ophthalmology and Visual Sciences, Center for Health Policy, Washington University School of Medicine, St Louis, Missouri: Steven M. Kymes, PhD; Department of Ophthalmology, Doheny Eye Center UCLA (University of California at Los Angeles), Arcadia: SriniVas R. Sadda, MD; University of Kansas Center for Telemedicine and Telehealth, Kansas City: Ryan Spaulding, PhD. National Eye Institute, Bethesda, Maryland: Eleanor B. Schron, PhD, RN.