ROPtool analysis of plus and pre-plus disease in narrow-field images: a multi-image quadrant-level approach

Abstract

Background

The presence of plus disease is important in determining when to treat retinopathy of prematurity (ROP), but the diagnosis of plus disease is subjective. Semiautomated computer programs (eg, ROPtool) can objectively measure retinal vascular characteristics in retinal images, but are limited by image quality. The purpose of this study was to evaluate whether ROPtool can accurately identify pre-plus and plus disease in narrow-field images of varying qualities using a new methodology that combines quadrant-level data from multiple images of a single retina.

Methods

This was a cross-sectional study of previously collected narrow-field retinal images of infants screened for ROP. Using one imaging session per infant, we evaluated the ability of ROPtool to analyze images using our new methodology and the accuracy of ROPtool indices (tortuosity index [TI], maximum tortuosity [T_max], dilation index [DI], maximum dilation [D_max], sum of adjusted indices [SAI], and tortuosity-weighted plus [TWP]) to identify pre-plus and plus disease in images compared to clinical examination findings.

Results

Of 198 eyes (from 99 infants) imaged, 769/792 quadrants (98%) were analyzable. Overall, 98% of eyes had 3–4 analyzable quadrants. For plus disease, area under the curves (AUCs) of receiver operating characteristic curves were: TWP (0.98) > TI (0.97) = T_max (0.97) > SAI (0.96) > DI (0.88) > D_max (0.84). For pre-plus or plus disease, AUCs were: TWP (0.95) > TI (0.94) = T_max (0.94) = SAI (0.94) > DI (0.86) > D_max (0.83).

Conclusions

Using a novel methodology combining quadrant-level data, ROPtool can analyze narrow-field images of varying quality to identify pre-plus and plus disease with high accuracy.

Retinopathy of prematurity (ROP) is a leading cause of preventable and treatable childhood blindness in the industrialized world.^1,2 Early treatment of high-risk ROP reduces unfavorable outcomes; thus, accurate screening for treatment-requiring ROP is imperative.³ The identification of plus disease is an integral component of diagnosing treatment-requiring (type 1) ROP.³ Plus disease is defined in comparison to a “standard” photograph that demonstrates the minimum amount of vascular dilation and tortuosity that must be present in at least 2 quadrants of the eye.^4,5 Pre-plus disease, defined as “vascular abnormalities of the posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arterial tortuosity and more venous dilation than normal,”⁵ has been shown to be predictive of the need for treatment.⁶

Because the diagnosis of plus disease is subjective, even among ROP experts,⁷ semiautomated computer programs have been developed to objectively measure retinal vascular tortuosity (eg, ROPtool, RISA, CAIAR, and iROP) and dilation (eg, ROPtool, RISA, CAIAR, and VesselMap).^8–12 Studies have shown that ROPtool can accurately identify pre-plus and plus disease by analyzing single retinal images.^8,13–20 ROPtool has been validated for assessing plus disease in wide-field (ie, RetCam; Natus Medical Inc, Pleasanton, CA^8,13–16) and narrow-field images (video indirect ophthalmoscopy [VIO]^17,18 and Pictor [Volk Optical Inc, Mentor, OH]^19,20). When analyzing VIO images of varying qualities, one study found that ROPtool could analyze only 24% of the images when evaluated on its ability to analyze 2 vessels in all 4 quadrants of the eye.¹⁷ Another study found that ROPtool was able to trace more images acquired by Pictor compared to VIO.²⁰ While one study found that among high-quality Pictor images, 95% were analyzable by ROPtool,¹⁹ it is unclear how ROPtool performs when analyzing Pictor images of varying qualities. The purpose of this study was to evaluate whether ROPtool can accurately identify pre-plus and plus disease in narrow-field Pictor images of varying qualities using a new methodology, which combines quadrant-level data from multiple images of a single retina.

Materials and Methods

The study was approved by the institutional review boards of Duke University Health System, Durham, North Carolina, and Cape Fear Valley Health System, Fayetteville, North Carolina, and complied with the regulations of the US Health Insurance Portability and Accountability Act of 1996.

Image Selection

This was a cross-sectional study of narrow-field (ie, Pictor) images previously collected as part of a prospective study where non-physician health care workers used Pictor to obtain retinal images of infants being screened for ROP.²¹ As part of this previous study, imagers selected 1–3 images obtained per eye at each imaging session for image grading. The goal of the image selection was that, when these images were viewed together, they would demonstrate vessels in all 4 quadrants in the posterior pole of the eye for telemedicine grading. For this study, one imaging session per eye per infant was chosen using a predefined selection algorithm that was used in a previous study to create an enhanced sample of images with disease.²² Basically, for each eye, if the eye received treatment, we chose the latest imaging session prior to treatment, and if the eye did not receive treatment, we chose the imaging session with the worst posterior pole disease (plus disease > pre-plus > neither) based on clinical diagnosis. If that included more than one imaging session, then we chose the session closest to a postmenstrual age (PMA) of 36 weeks. If two imaging sessions were performed equidistant from a PMA of 36 weeks, the earlier of the two was chosen.

ROPtool Analysis

A non-ophthalmologist (MCW), who was masked to the clinical diagnosis, performed all ROPtool (v2.1.8) analyses. For each eye, she examined the 1–3 images submitted for the selected imaging session to determine which retinal quadrants would be analyzed in each image to include all 4 retinal quadrants. For a particular quadrant, she chose the image with highest clarity and longest vessel length for analysis. In ROPtool, she created and assigned quadrants by placing a crosshair centered on the optic nerve and attempting to include 2 major retinal vessels in each quadrant (Figure 1). Quadrants were maintained among the images, preventing duplicate tracing of a vessel in multiple quadrants. For each quadrant, ROPtool was used to analyze 1–2 vessels by selecting vessels to trace using the following algorithm: (1) trace the most tortuous major vessel, (2) trace the most dilated major vessel, and (3) when the most tortuous and/or most dilated vessels are not traceable, trace the major vessel with the longest traceable portion. A vessel was considered “traceable” by ROPtool when it could be traced for a length of at least 1 optic disk diameter. A quadrant was considered “analyzable” when it contained at least 1 traceable vessel. Using ROPtool, the following indices were obtained for each quadrant:

• Tortuosity index (TI). Vessel tortuosity was determined by measuring the total length of a vessel and comparing it to the length of a smooth curve generated from points equally spaced along the vessel.¹⁴ TI was defined as the vessel tortuosity divided by the average tortuosity of the standard photograph of plus disease,⁴ multiplied by 10, and was the result of averaging these values for all vessels in a quadrant.¹⁴

• Maximum tortuosity (T_max). T_max was defined as the maximum vessel TI in that quadrant.

• Dilation index (DI). Vessel dilation was determined by averaging the widths of multiple cross-sections of a vessel and dividing by the distance from the center of the optic nerve to the center of the macula to account for image magnification.¹⁴ DI was defined as the vessel dilation divided by the average dilation of the standard photograph of plus disease,⁴ multiplied by 10, and was the result of averaging these values for all vessels in a quadrant.¹⁴

• Maximum dilation (D_max). D_max was defined as the maximum vessel DI in that quadrant.

• Sum of adjusted indices (SAI). The adjusted tortuosity index (ATI) compresses the TI to a value more comparable to the DI. For a TI <10, ATI = 10 − 2/3 (10 − TI). For a TI >10, ATI = 10 + (TI − 10)/10.⁸ The SAI then sums the ATI and the DI for a quadrant.⁸

• Tortuosity-weighted plus (TWP). The TWP measurement was based on the observation that tortuosity may influence clinicians’ judgment of plus disease more than dilation does. TWP = DI × ATI/12 + ATI.⁸

FIG 1.

Quadrant creation and labeling using ROPtool. ROPtool is a semiautomated computer program that requires user input. The user begins by outlining the optic nerve (yellow circle) and selecting the location of the approximate center of the macula (brown dot). ROPtool generates a crosshair including both a horizontal line that connects the center of the optic nerve and the center of the macula and a vertical line, delineating the 4 quadrants (A). These quadrant lines can be adjusted by the user to adjust for vessel origin and ensure at least 1 major blood vessel is included in each quadrant. Quadrant labeling method: the upper left quadrant is labeled A, upper right B, lower left C, and lower right D. Example of defining and labeling quadrants in the right eye (B) and left eye (C).

Each quadrant was assessed independently and then the data was combined for eye-level analysis (Figure 2).

FIG 2.

ROPtool analysis of an eye using multiple narrow-field retinal images obtained from the same imaging session. a: ROPtool analysis performed on a single quadrant of a left eye. Measurement indices computed for the quadrant outlined by blue box. b: Images demonstrating quadrant-level analysis of the same left eye. Three separate images were used to trace vessels in all 4 quadrants. Two vessels were traced in each quadrant. c: Quadrants of images from b montaged together to render the appearance of this single retina.

Using the area under the curve (AUC) measurements of receiver operating characteristic (ROC) curves, we evaluated the ability of ROPtool indices of tortuosity, dilation, and a combination of the two to identify pre-plus and plus disease in the images compared to the clinical examination findings (ie, the reference standard). All clinical examinations were performed by an ophthalmologist (SFF, JWR, DKW) with at least 15 years’ experience in ROP screening.²¹ ROC curves and AUC measurements were generated using JMP Pro (v12.0.1, SAS Institute Inc, Cary, NC) for the identification of (1) plus disease, and (2) pre-plus or plus disease, using the second-largest quadrant value of each ROPtool index for those eyes with at least 3 analyzable quadrants. The second largest value was chosen based on the definition of plus disease requiring sufficient vascular tortuosity and dilation present in at least 2 quadrants.^4,5 Eyes with ≤2 analyzable quadrants were excluded from the analysis because a diagnosis of plus disease could not be precluded when the remaining analyzable quadrants were considered normal. The identification of pre-plus disease was explored because if only 3 quadrants were analyzable, and the unanalyzable quadrant was one of 2 quadrants with plus disease, then the eye would at minimum be classified as pre-plus.

Results

This study included images acquired from 99 infants (198 eyes). Of the eye imaging sessions selected, 11 eyes (6%) had pre-plus disease and 9 (5%) had plus disease as determined on clinical examination. Of the images submitted for the 198 eyes imaged, 769/792 quadrants (98%) were analyzable by ROPtool, meaning they had at least 1 vessel traceable for at least 1 optic disk diameter in length. Of 198 eyes, 179 (90.4%) had 4 analyzable quadrants, 15 (7.6%) had 3 analyzable quadrants, and 4 (2.0%) had 2 analyzable quadrants. All eyes had at least 2 analyzable quadrants. When stricter criteria were applied, requiring at least 2 vessels traceable for at least 1 disk diameter in length per quadrant, 136 eyes (68.7%) had 4 quadrants with 2 traceable vessels, 37 (18.7%) had 3 quadrants, 17 (8.6%) had 2 quadrants, 3 (1.5%) had 1 quadrant, and 5 (2.5%) had 0 quadrants. There were 692/792 quadrants (87%) with at least 2 traceable vessels.

When evaluating the ability of ROPtool indices to identify pre-plus and plus disease in images compared to clinical examination findings, 194/198 (98%) eyes were included in analysis. Four eyes were excluded from the analysis as they had only 2 analyzable quadrants. ROC curves were generated and the AUC calculated for the identification of (1) plus disease and (2) pre-plus or plus disease (Figure 3). All tortuosity or combination measures had an AUC of ≥0.96 for identifying plus disease and ≥0.94 for identifying pre-plus or plus disease (Figure 3).

FIG 3.

Receiver operating characteristic (ROC) curves of ROPtool analysis. This includes dilation (dilation index [DI], maximum dilation [D_max]), tortuosity (tortuosity index [TI], maximum tortuosity [T_max]), and combination (sum of adjusted indices [SAI], tortuosity-weighted plus [TWP]) indices for the identification of (1) plus disease and (2) pre-plus or plus disease. Area under the curve (AUC) measurements are included in each legend.

Discussion

In this study, ROPtool, using a new methodology combining quadrant-level data from multiple images, was able to analyze narrow-field images of varying quality to identify (1) plus disease and (2) pre-plus or plus disease with high accuracy. By combining quadrant-level analysis from multiple narrow-field images, we achieved high levels of analyzability compared with previous ROPtool studies, which were based on analyses of single images from wide- and narrow-field images (Table 1).^16,17,19,20 Compared with previous studies, in which ROPtool was used to analyze narrow-field Pictor images, we found high traceability/analyzability (a marker for image quality), despite inclusion of images of varying quality; our inclusion of all images—not only those of high quality—simulated a real-life screening scenario (Table 1).^19,20 We found that 98% of quadrants had at least 1 traceable vessel, and 90% of eyes had at least 1 traceable vessel in all 4 quadrants by ROPtool. A unique feature of this study was quadrant-level analysis. In allowing the use of multiple narrow-field images of the same retina in ROPtool analysis, quadrant-level analysis improved the ability of ROPtool to analyze images of varying quality for the assessment of both (1) plus disease and (2) pre-plus or plus disease with similar or greater accuracy as only using high-quality images.

Table 1.

Comparison of ROPtool studies: overall ability of ROPtool to analyze still images

Study	Johnston et al16	Ahmed et al17	Vickers et al19		Raufi et al20		Current study
Type of image	RetCam	VIOa	Pictor		VIOa	Pictor	Pictor
Image field of view	Wide	Narrow	Narrow		Narrow	Narrow	Narrow
Image quality	High	Varying	Varying	High	High	High	Varying
Eyes with ≥1 vessel traceable for ≥1 disk diameter length in 4 quadrants	NR	NR	35/48 (73%)	35/37 (95%)	13/23 (57%)	21/23 (91%)	179/198 (90%)
Eyes with ≥2 traceable vessels in 4 quadrantsb	77/179 (43%)	6/25 (24%)	NR	NR	NR	NR	136/198 (69%)
Quadrants with ≥2 traceable vessels	540/716 (75%)	43/100 (43%)	NR	NR	NR	NR	692/792 (87%)

NR, not reported; VIO, video indirect ophthalmoscopy.

^aVIO still images are taken from videos acquired using a VIO system during the clinical examination.

^bDefinitions vary per study: Johnston et al¹⁶ could trace ≥1 arteriole and ≥1 venule for ≥1 optic disk diameter; Ahmed et al¹⁷ and the current study could trace ≥2 vessels for ≥1 optic disk diameter.

Our new methodology improved ROPtool’s ability to analyze narrow-field images, which, in turn, improved its diagnostic accuracy of plus disease for narrow-field images. By including images of varying quality and a larger sample size, we were able to build on the results of our previous studies.^19,20 When using ROPtool as a screening tool, if images have at least 3 analyzable quadrants, we can effectively rule out plus disease if all analyzable quadrants are within normal limits. With identification of the presence of pre-plus or plus disease, we can efficiently screen-in infants requiring a complete ophthalmic examination. Of the various ROPtool indices explored, retinal tortuosity was the most essential component in the identification of pre-plus or plus disease. Of the six ROPtool indices assessed, four —TI, T_max, SAI, and TWP—had an AUC of ≥0.96 for the identification of plus disease and ≥0.94 for the identification of pre-plus or plus disease (Figure 3). Notably, these parameters are tortuosity or combination measurements, reinforcing the notion that tortuosity is the driving factor behind the clinical diagnosis of plus disease.⁸ Furthermore, TWP, the measure designed to most closely reflect how we believe clinicians judge plus disease,⁸ was superior for identifying plus disease and plus or pre-plus disease. Our results correlate with prior research using CAIAR and RISA, which, respectively, found AUCs of 0.90 and 0.82 for venule width, 0.79 and 0.74 for arteriolar tortuosity, and 0.96 and 0.97 for a combination of width and tortuosity measures, further supporting that combination measures may have the highest degree of clinical accuracy.^9,10

There have been exciting advances in automated grading for ROP screening using artificial intelligence (AI) and deep learning (DL), and study results have been favorable with AUC = 0.98 for the identification of plus disease.^23,24 These technologies are built from large databases of reference images for the algorithm to “learn” how to identify a disease process. Most databases are built from wide-field RetCam images.^23–25 RetCam is an expensive, large, wide-field, contact camera. Our study demonstrates the ability to use images acquired from a less expensive, portable, narrow-field noncontact camera for screening. We unfortunately lack the requisite large number of narrow-field retinal images for AI and DL at this time but found that our method has high sensitivity and specificity for screening for type 1 ROP. Our new methodology could be used with other vessel analysis programs, AI, and DL to improve the accuracy of plus disease diagnoses, particularly if narrow-field imaging is used. Moving forward, we believe there is a role for both types of technology.

Our study had some limitations. First, our reference standard is the clinical diagnosis of pre-plus or plus disease, which has been shown to be subjective, even among ROP experts.⁷ Admittedly not as rigorous as multiple expert review, our clinical examiners (DKW, SFF, JWR) nevertheless each have at least 15 years’ experience in ROP screening. Although we selected only a single clinical examination for each eye per infant, our goal was to select a sample enhanced with posterior pole disease. We chose to define “traceable vessel” length as 1 disk diameter. Even though this is a relatively short segment, we felt this is the minimal length of a major vessel needed to be able to make some assessment of its dilation and tortuosity, and we wanted to be able to compare our results to previous studies that used this same definition.^8,14,17–19

This study validates the potential applicability and high diagnostic yield of narrow-field imaging in ROP screening; however, additional steps are needed before a combination of Pictor imaging and ROPtool analysis can be used as a screening tool. One study has used ROPtool to longitudinally evaluate VIO images and found a statistically significant difference in the highest mean tortuosity indices between infants eventually requiring treatment and those not requiring treatment.²⁶ Because our new methodology is now validated for the identification of plus and pre-plus disease, our next step will be to validate ROPtool using this methodology in a longitudinal study of Pictor images to determine which indicators could help predict which infants may progress to needing treatment. Also, given the promising results using our new methodology of combining quadrant-level analysis from multiple narrow-field images, the ability to analyze a picture montage could further simplify and make more efficient this methodology.²⁷ Additionally, as more data is collected, cut-off values for vessel tortuosity and dilation will need to be established to enable ROPtool to diagnose pre-plus and plus disease. Currently, ROPtool is a semiautomated program that requires human user input, but its complete automation could increase the objectivity of analysis and make the determination of pre-plus and plus disease less subjective in determining which infants need to be examined by an ophthalmologist.