Evaluation of an indirect ophthalmoscopy digital photographic system as a retinopathy of prematurity screening tool

Abstract

Purpose

To determine whether digital retinal images obtained from an indirect ophthalmoscopy imaging system (Keeler) can be accurately graded for clinically significant retinopathy of prematurity (ROP) by masked experts.

Methods

The medical records of infants screened for ROP who had posterior pole images acquired using the Keeler system during routine ROP examinations were retrospectively reviewed. Two reviewers, masked to patient demographics and clinical examination findings, graded the images for (1) quality (good, fair, poor); (2) number of gradable quadrants, from 0 to 4; and (3) posterior pole disease (none, pre-plus, plus). The accuracy of grading Keeler images for clinically significant ROP (defined as pre-plus or plus disease) was compared to results of clinical examination.

Results

One eye each of 253 infants was included. The mean postmenstrual age at examination was 35 weeks (range, 30–42). Grader 1 found the quality of 94% of images to be fair or good; grader 2, 83% of images. Grader 1 judged 87% of images to have ≥3 gradable quadrants; grader 2, 77% of images. The sensitivity and specificity of grading pre-plus or worse disease on Keeler images were 100% and 86%, respectively, for grader 1, and 94% and 89%, respectively, for grader 2.

Conclusions

Digital retinal images obtained by the Keeler system can be read with high sensitivity and specificity to screen for clinically important ROP. The Keeler system may be a valuable tool for ROP screening at remote locations (ie, via telemedicine).

Retinopathy of prematurity (ROP) remains an important cause of blindness, especially in the developing world.¹ Appropriate screening and treatment could reduce the burden of childhood blindness due to ROP, but there are many barriers to effective ROP screening, including the shortage of ophthalmologists trained to screen for ROP² and the lack of access to these ophthalmologists. According to current guidelines in the United States, retinal screening examinations should be performed by an ophthalmologist trained to screen for ROP using binocular indirect ophthalmoscopy with a lid speculum, with or without scleral depression.³

The Vantage Plus LED Digital Binocular Indirect Ophthalmoscope system (Keeler Instruments Inc, Broomall, PA) consists of a binocular indirect ophthalmoscope with an integrated camera that can capture still and/or dynamic images during the examination, which can be stored in various digital formats for later review. The field of view obtained by the Keeler system is similar to that seen during the standard examination with binocular indirect ophthalmoscopy. The field of view of images obtained by the Keeler system should theoretically be adequate for evaluating the posterior pole for the presence of pre-plus or plus disease.

Currently, the presence of plus disease drives the decision to treat ROP.⁴ In the absence of plus disease, type 1 ROP (ie, treatment-requiring ROP) can be present only if there is stage 3 in zone 1, which is not only unusual but also would not be expected with a completely normal posterior pole.⁵ Thus it might not be necessary for a true ROP “screening test” (versus a diagnostic examination performed by a trained ophthalmologist) to include views of the peripheral retina if the objective is to identify infants requiring a standard examination with indirect ophthalmoscopy by an experienced ophthalmologist to evaluate need for treatment. Obtaining and interpreting images of the vessels of the posterior pole alone may be a reasonable method to screen for those infants with type 1 ROP. The purpose of this study was to evaluate whether digital retinal images obtained using an indirect ophthalmoscopy imaging system could be accurately graded by masked experts for clinically significant ROP (CSROP), defined for purposes of this study as pre-plus or plus disease.

Methods and Materials

This study was approved by the Duke Health System Institutional Review Board and conformed to the requirements of the US Health Insurance Portability and Accountability Act of 1996. The medical records of all infants screened for ROP over a 2-year period (November 2009-November 2011) at the Duke University Neonatal Intensive Care Unit (NICU) were retrospectively reviewed. Demographic data of eligible patients were extracted, including date of birth, gestational age, birth weight, and date of ROP examinations. Postmenstrual age was calculated based on date of examination and date of birth. As part of our routine screening for ROP, we digitally recorded every examination using the Vantage Plus LED Digital Binocular Indirect Ophthalmoscope and a 28 D condensing lens. All examinations were performed by one of two pediatric ophthalmologists (SFF or DKW), both of whom have extensive experience with ROP examination and classification and have been certified investigators in multicenter ROP clinical trials.^4,6,7 Prior to examination, all infants were dilated. At our institution, ROP examinations occurred starting at 30 weeks postmenstrual age or 4 weeks of age, whichever was later, for infants with birth weight <1500 g or gestational age ≤30 weeks, and for selected infants with birth weight 1500–2000 g or gestational age >30 weeks who had an unstable clinical course, per recommended guidelines at the time of screening.⁸ Follow-up examinations occurred according to current published guidelines at the time of the examination.^8,4 The presence or absence of ROP and the zone, stage, and presence or absence of plus or pre-plus disease were documented for each eye according to current international classification guidelines.⁷ Our criterion for laser treatment was the development of type 1 ROP as established by the Early Treatment for ROP study.⁴

Inclusion criteria included hospitalization at the Duke University NICU, birth weight <1500 g or gestational age ≤30 weeks, ROP screening from November 1, 2009 to November 16, 2011, and availability of digital images obtained by the Keeler system at the time of screening. Infants were excluded if they had received laser or anti-vascular endothelial growth factor (VEGF) treatment prior to having an examination recorded by the Keeler system during the study period.

The images from one examination date were chosen for each infant (Figure 1). The images were chosen in order to enhance the sample by including an adequate number of posterior pole images representing pre-plus and plus disease. If an infant required treatment (ie, laser or anti-VEGF treatment), the latest eligible examination date prior to treatment was selected. Otherwise, for each infant, the examination date with the most severe posterior pole disease (plus > pre-plus > normal) was selected. If there were several examination dates that contained the most severe posterior pole disease, the date closest to when the infant had a postmenstrual age of 36 weeks was selected. If two examinations were performed equidistant from 36 weeks, the earlier examination date was chosen. Any examination performed after treatment was excluded.

FIG 1.

Image selection strategy. PMA, postmenstrual age; ROP, retinopathy of prematurity; VEGF, vascular endothelial growth factor.

After the examination date was chosen, we reviewed the images recorded on that date by the Keeler system for the infant. If a video recording was obtained on the selected screening date, still images were created using a video converter (Windows Movie Maker 2.6, Microsoft, Redmond, WA). To be eligible for inclusion, images had to include a view of the optic nerve. Only images from one eye for each infant were included. Images of the right eye were selected unless no eligible image was available, in which case left eye images of the same examination date were selected. Up to 3 images for the selected eye could be included because not all of the images of the posterior pole were centered on the optic nerve and our goal was to provide graders with at least 1 disk diameter length of a major vessel in each quadrant.

An electronic slide show was created with one “unknown” image per slide. If more than 1 image was included per subject, these images were placed in the slide show consecutively and labeled A, B, and C, as appropriate, to indicate that the images belonged to the same subject (up to 3 slides for each eye/infant). These slides, along with a sample of repeat images, were randomly placed in a slide show without any demographic or clinical information for the graders. Twenty repeat slides were randomly selected from all slides proportional to the number of those with normal, pre-plus, and plus disease in the original study population in order to assess intra-grader reliability. Graders were provided with standard reference images for pre-plus and plus disease from the International Classification of ROP (ICROP) revisited paper,^7,9 cropped to display a field of view similar to that captured by the Keeler system through a 28 D lens and with similar magnification to the study images. The standard photographs and the study images were displayed on the same interface to allow for direct comparison (Figure 2).

FIG 2.

A, Slide with an “unknown” image and standard photographs of pre-plus and plus disease displayed on the same interface to allow for direct comparison. B, An additional reference slide presented to the graders to show the range of disease for pre-plus and plus disease. Images are reproduced and cropped to display a field of view similar to that of Keeler images for comparison. Reprinted with permission. International Committee for the Classification of Retinopathy of Prematurity. The International Classification of Retinopathy of Prematurity revisited. Arch Ophthalmol 2005;123:991–9. Capone A Jr, Ells AL, Fielder AR, et al. Standard image of plus disease in retinopathy of prematurity. Arch Ophthalmol 2006;124:1669–70. Copyright ©2005 and 2006, American Medical Association. All rights reserved. Personal use of this material is permitted. However, permission to reuse this material for any other purpose must be obtained from the American Medical Association.

Two ROP experts masked to demographic information and clinical findings (SFF and DKW) independently reviewed the slide show and evaluated the images for (1) quality, (2) number of gradable quadrants, and (3) posterior pole disease. Based on the ability of the grader to determine the dilation and/or tortuosity of the vessels in all images selected from one examination date for each infant, image quality was graded as follows: “good,” providing a clear view of both the optic nerve and vessels such that the grader could easily discern the dilation and tortuosity of the vessels; “fair,” in which either dilation or tortuosity was difficult to discern; or “poor,” in which both dilation and tortuosity could not be clearly discerned. The number of gradable quadrants (0 to 4) was scored based on the visibility of at least 1 disk diameter length of a major vessel in a given quadrant. Posterior pole disease was graded as (1) normal, (2) pre-plus, or (3) plus disease. In this study, pre-plus disease was defined according to the ICROP revisited definition of “vascular abnormalities of the posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arterial tortuosity and more venous dilatation than normal,”⁷ and plus disease was defined as the presence in ≥2 quadrants of the eye of sufficient vascular dilation and tortuosity as compared to a standard photograph.^7,10 Because both graders performed ROP screening during the period of image acquisition (each for 50% of the time period), they pledged to recuse themselves from grading any images that they recognized.

SAS 9.3 (SAS Institute Inc, Cary, NC) was used for all statistical analysis. Prior to the commencement of this study, a sample size calculation indicated that, assuming a sensitivity of 0.85, a sample size of 204 was required to generate a one-sided confidence interval width of 0.05. In order to appropriately power our study to detect a sensitivity of 0.85 (95% CI, 0.80–0.90), we included all infants screened for ROP in the Duke University NICU over a 2-year period who fulfilled study inclusion criteria. Before analyzing the data, we defined the “reference standard” as the diagnosis of plus disease by indirect ophthalmoscopy during the clinical examination. We defined CSROP as the presence of pre-plus or plus disease. We chose to include pre-plus in this definition because we were willing to sacrifice specificity in order to have very high sensitivity for detection of plus disease so as not to miss treatment-requiring cases of ROP. For the primary analysis of accuracy, we evaluated the ability of each grader to accurately identify CSROP on Keeler images compared to the “reference standard.” A secondary analysis for accuracy compared the grading of Keeler images for pre-plus or worse disease to the diagnosis of pre-plus or worse disease by indirect ophthalmoscopy during the clinical examination on the same infant at the same screening session.

Results

A total of 253 infants were included (average gestational age, 27 weeks [range, 23–34]; average birth weight, 961 g [range, 450–2300 g]; average postmenstrual age at examination, 35 weeks [range, 30–42]). For 1 infant there was no eligible image of the right eye on the selected examination date, and left eye images of the same date were used. As diagnosed by indirect ophthalmoscopy during the clinical examination, our enhanced sample of images was composed of 7% with plus disease, 13% with pre-plus disease, and 80% with a normal posterior pole.

Overall, grader 1 found image quality to be good or fair in 94% of images; grader 2, in 83% of images (Table 1). Grader 1 judged images as having ≥3 gradable quadrants in 87% of images; grader 2, in 77% of images (Table 2). Neither of the graders recognized any of the images used in this study and thus did not need to recuse themselves from grading any of the images.

Table 1.

Image quality of Keeler fundus photographs

	Number of images by qualitya (%)
	Good	Fair	Poor
Grader 1	164 (65)	74 (29)	15 (6)
Grader 2	117 (46)	93 (37)	43 (17)

^aBased on all images selected from one examination date for each infant. Quality was graded as follows: “good,” providing a clear view of both the optic nerve and vessels such that the grader could easily discern the dilation and tortuosity of the vessels; “fair,” in which either dilation or tortuosity was difficult to discern; or “poor,” in which both dilation and tortuosity could not be clearly discerned.

Table 2.

Gradable quadrants of Keeler fundus photographs

	Number of gradable quadrantsa (%)
	4	3	2	1	0
Grader 1	179 (71)	40 (16)	21 (8)	9 (4)	4 (2)
Grader 2	156 (62)	39 (15)	25 (10)	23 (9)	10 (4)

^aBased on the adequate visibility of at least 1 disk diameter length of a major vessel in a given quadrant based on all images selected from one examination date for each infant.

Using the reference standard of plus disease identified by indirect ophthalmoscopy, the sensitivity of grading CSROP (pre-plus or plus disease) for grader 1 and grader 2 was 100% and 94%, respectively, and the specificity of grading CSROP for graders 1 and 2 was 86% and 89%, respectively (Table 3). Of the cases of clinically diagnosed plus disease by indirect ophthalmoscopy, grader 1 graded all cases as either pre-plus or plus, whereas grader 2 judged 1 case of clinically diagnosed plus disease to be normal (Table 3). On further evaluation of the images for this case, both graders felt that only 2 quadrants were gradable and whereas grader 1 judged the image quality as fair, grader 2 felt image quality was poor.

Table 3.

Accuracy of grading posterior pole disease on Keeler fundus photographs compared to reference standard

		Reference standarda
		Plusb	Pre-plusc	Normal
Grader 1	Plusb	6	3	1
	Pre-plusc	11	20	8
	Normal	0	10	190
	Don’t know	0	0	4
Grader 2	Plusb	11	2	0
	Pre-plusc	5	19	6
	Normal	1	12	197
	Total number (%)	17 (7)	33 (13)	203 (80)

^aGrading of posterior pole disease by indirect ophthalmoscopy on the clinical examination.

^bImage(s) with at least 2 quadrants of the eye with sufficient vascular dilation and tortuosity present as compared to a standard photograph.^7,10

^cImage(s) with “vascular abnormalities of the posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arterial tortuosity and more venous dilatation than normal.”⁷

Using the reference standard of pre-plus or plus disease identified by indirect ophthalmoscopy, the sensitivity of grading CSROP (pre-plus or plus disease) for grader 1 and grader 2 was 80% and 74%, respectively, and the specificity of grading CSROP for graders 1 and 2 was 95% and 97%, respectively (Table 3).

Intra-grader reliability for grading CSROP was 85% (κ = 0.54) for grader 1 and 100% (κ= 1.00) for grader 2. Inter-grader reliability was 95% (κ= 0.85) for CSROP on Keeler fundus images (Table 4, Figure 3).

Table 4.

Inter-grader reliability of grading posterior pole disease in Keeler fundus photographs

		Grader 2
		Plusa	Pre-plusb	Normal	Total number (%)
Grader 1	Plusa	6	4	0	10 (4)
	Pre-plusb	7	25	7	39 (15)
	Normal	0	1	199	200 (79)
	Don’t know	0	0	4	4 (2)
	Total number (%)	13 (5)	30 (12)	210 (83)	253 (100)

^aImage(s) with at least 2 quadrants of the eye with sufficient vascular dilation and tortuosity present as compared to a standard photograph.^7,10

^bImage(s) with “vascular abnormalities of the posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arterial tortuosity and more venous dilatation than normal.”⁷

FIG 3.

Representative Keeler images in which the graders both agreed showed a normal posterior pole (A), pre-plus (B), and plus disease (C).

Discussion

In this study, retinal images captured during routine ROP screening using the Keeler system were of sufficient quality to accurately grade for the presence of CSROP in premature infants who were at risk for ROP. Graders evaluated these retinal images for the presence of pre-plus or worse disease to determine whether infants would pass or fail the “screening test” for CSROP, which would (in this scenario) trigger a standard diagnostic examination by an experienced ophthalmologist using indirect ophthalmoscopy. We chose the presence of pre-plus or plus disease as the cut-off for failing the “screening test” because the presence of plus disease currently drives the decision to treat ROP.⁴ In the absence of plus disease, type 1 (ie, treatment-requiring) ROP is present only if there is stage 3 in zone 1.

Although we did not have graders look for the presence of stage 3, the presence of pre-plus or worse disease is a reasonable surrogate for significant ROP in the periphery. One study found that normal posterior pole vessels are a reliable marker for the absence of stage 3 ROP.⁵ Thus a true ROP “screening test” may not need to include views of the peripheral retina if the objective is to screen for infants who require a standard examination with indirect ophthalmoscopy by an experienced ophthalmologist to evaluate the need for treatment. Also, the presence of pre-plus disease has been shown to be highly associated with the development of treatment-requiring ROP and to contribute further prognostic value regarding the future need for laser therapy beyond that already offered by birth weight, gestational age, ROP zone, and ROP stage.¹¹ Thus we propose that infants would fail the “screening” if the images obtained by the Keeler system showed the presence of pre-plus or worse disease. This finding would, in turn, trigger the need for a standard binocular indirect ophthalmoscopy examination by an ophthalmologist trained in ROP screening; the examination would not only reevaluate the presence of posterior pole disease but also determine the zone and stage of ROP in the retinal periphery.

The grading of retinal images acquired by the Keeler system for pre-plus or worse disease showed a high sensitivity and specificity compared to the clinical examination, suggesting that the Keeler system shows promise as an ROP screening tool. A good “screening test” for treatment-requiring ROP must have a high sensitivity, so that those with disease are not missed. High specificity is also desirable so that those without disease are not subjected to superfluous examinations, which may be invasive or stressful. Of the 17 cases of clinically diagnosed plus disease by indirect ophthalmoscopy, 1 case (6%) was judged to be normal by photographic review (Table 3, grader 2); the image was judged by that grader to be of poor quality and have only 2 gradable quadrants. Thus if screening criteria using the Keeler system required the visualization of at least 3 quadrants and image quality to be fair or good, this infant’s original images would have failed screening criteria. Hence the infant would have either been reimaged or merited an examination by an ophthalmologist trained in ROP screening, making it unlikely that the infant would have developed treatment-requiring ROP that was missed.

Both graders showed both high intra- and inter-grader reliability for grading pre-plus or worse disease. While previous studies have attained lower inter-expert agreement of diagnosing plus^12,13 and pre-plus disease,¹³ this may be because of differences in the severity of disease included in the sample of images evaluated. In a study by Chiang and colleagues,¹² the images graded were chosen from those that showed a change in characteristics of the vasculature compared to baseline. In another study, an enriched image set was created by including a larger proportion with plus and pre-plus disease than would normally be encountered during routine ROP screening.¹³ In the present study, primary analysis was based on the agreement of two categories (ie, cases with pre-plus and plus disease were in the same diagnosis group) instead of three categories, which improved the agreement between our graders. We also intentionally included standard images from ICROP revisited, providing examples of pre-plus and plus disease cropped to a similar magnification to the unknown images to allow direct comparisons to be made, which may have further improved agreement between the graders.

The Keeler system offers many qualities desirable in a screening tool. If used with a laptop computer, the Keeler system is small and portable between patients or clinical locations. While indirect ophthalmoscopy is a skill that all graduating ophthalmologists are expected to master during residency, ROP screening is not routinely taught to ophthalmologists-in-training.¹⁴ Even among pediatric and retinal fellowship training programs in the United States, exposure to ROP screening widely varies across institutions.¹⁵ Theoretically, any trained ophthalmologist should be able to use the Keeler system to capture images of the posterior pole during an indirect ophthalmoscopy examination to be reviewed at a later time. Thus using the Keeler system to image the posterior pole of infants at risk for ROP, with remote image review by experts, could increase the number of ophthalmologists involved in ROP screening, reducing the burden of screening on the currently limited number of ROP experts.

The present study had several limitations. At our institution, it is standard practice to proceed with the ROP evaluation despite poorly dilated pupils if the examiner can adequately rule out severe ROP, so infants did not necessarily receive extra dilation drops if their pupils were small. Thus the quality of images evaluated in this study likely underestimates the quality of images obtainable if the acquisition of images for ROP screening (rather than the clinical examination) were the goal or if a prospective study was done to specifically assess the ability to obtain images. In addition, all images were acquired from infants in a single NICU, with both graders also from the same institution. A multicenter study would increase the number of infants included and having graders from multiple sites evaluate the images would help determine the range of agreement between graders and across institutions for evaluating images captured by Keeler system for CSROP. Finally, it is possible that study results were influenced by the selection of particular images from an existing set. However, our image selection strategy was conceived to maximize the presence of posterior pole pathology in our sample set and evaluate images taken at an age when CSROP tends to occur. Using an enhanced sample enabled us to more thoroughly assess graders’ ability to evaluate the presence of posterior pole disease (ie, positive predictive value).

Alternative methods to truly “screen” for ROP are needed not only to decrease the burden of performing diagnostic examinations on the limited number of ophthalmologists trained to screen for ROP but also facilitate wider access to screening for at-risk infants. To address the relative shortage of qualified ROP screeners, many studies have turned to retinal imaging and telemedicine as a possible solution.^16–19 The Keeler system may be a valuable tool not only for ROP screening at a distance (ie, via telemedicine) but also for recording images to help educate ophthalmologists in the nuances of posterior pole evaluation in ROP screening. While using the Keeler system would likely not be difficult for an ophthalmologist, a nonophthalmologist would need significant training with indirect ophthalmoscopy prior to using the Keeler system. One study found that with sufficient training, nonophthalmologist physicians could use an indirect ophthalmoscope to reliably detect posterior pole retinal vascular changes.²⁰ Thus, the use of nonophthalmologists to help “screen” for ROP using the Keeler system may be feasible.

This study suggests that the Keeler system shows promise as an ROP screening tool. Prospective studies are warranted to evaluate the use of the Keeler system both to screen for ROP as well as to teach aspiring ROP screeners.