Computer versus human-expert ranking of posterior pole vascular tortuosity and dilation using retinal vessel maps generated from bedside optical coherence tomography: a proof-of-concept study

Abstract

Semiautomated computer software (ie, ROPtool) can trace and analyze optical coherence tomography (OCT)–generated retinal vessel maps for plus/pre-plus disease with high reliability and accuracy. This proof-of-concept study found that ROPtool can reliably rank OCT-generated vessel maps for tortuosity and combined tortuosity/dilation, which correlated well with human-expert rankings and clinical examination.

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide, and at-risk preterm infants require screening for the development of treatment-warranted ROP.^1,2 Plus disease (redefined by the International Classification of ROP, Third Edition, as retinal vascular tortuosity and dilation in zone I) is an important feature of most treatment-warranted ROP, but its diagnosis is subjective and has low intergrader agreement, even among experts.^3,4 An explanation for low interobserver agreement is that plus and pre-plus disease lie along a continuum of vascular tortuosity and dilation.⁴

High-contrast grayscale en face retinal vessel maps, generated from bedside optical coherence tomography (OCT), can be reliably ranked by human experts for ROP vascular disease severity and reliably and accurately traced and analyzed for the presence of plus or pre-plus disease by a semiautomated vessel analysis program (ie, ROPtool).^5,6 It is unknown whether ROPtool can reliably rank vascular disease severity from OCT-generated vessel maps, and how these rankings correlate with those of human ROP experts. The purpose of this proof-of-concept study was to evaluate the correlation between ROPtool and human ROP experts in ranking OCT-generated retinal vessel maps by increasing vascular tortuosity and/or dilation.

Methods

This study analyzed OCT images captured prospectively by an investigational high-speed, ultra-compact, handheld, non-contact, swept-source OCT (“UC3”) from awake preterm infants under the STudy of Eye imaging in Premature infantS (BabySTEPS; NCT02887157), where infants were eligible for enrollment if they met ROP screening criteria (ie, birthweight ≤1500g or gestational age ≤30 weeks).^2,7 This sub-study was approved as part of BabySTEPS by the Duke University Health System Institutional Review Board and conformed to the requirements of the US Health Insurance Portability and Accountability Act of 1996. Each participant’s parent/guardian provided written informed consent. For this sub-study, we included only infants with OCT images that captured all four posterior pole quadrants of both eyes.⁶ We excluded imaging sessions that occurred after ROP treatment. We selected one imaging session/eye, using a pre-determined algorithm designed to maximize the number of eyes with plus/pre-plus disease.⁸ For each imaging session, we created two-dimensional en face vessel maps by extracting the maximum pixel intensity at each A-scan position. We included up to 4 OCT vessel maps per imaging session to capture all four posterior pole quadrants.

For ROPtool analysis, two tracers (nonophthalmologists masked to human-expert rankings and clinical examination results) used ROPtool (v2.1.8, Duke University, Durham, NC) to trace and analyze ≤4 vessel maps per eye (Figure 1). ROPtool calculated the following vascular tortuosity and/or dilation indices for each quadrant: tortuosity index (TI, total vessel length divided by the length of a curve generated from equally spaced points along the vessel); dilation index (DI, average width of multiple cross-sections of the vessel); and sum of adjusted indices (SAI, sum of DI and TI, in which both are weighted equally).⁸ Eye-level ROPtool indices were calculated using quadrant-level methodology, which combines quadrant-level measures from multiple images into a single eye-level measure for each eye.⁸ For each ROPtool index, the vessel maps were ranked in order of increasing index value.

FIG 1.

ROPtool tracings of optical coherence tomography–generated retinal vessel maps of preterm infant eyes. Retinal vessel maps were montaged to create a single vessel map per eye that captured all four posterior pole quadrants. The tracer uploads a single vessel map into ROPtool, outlines the optic nerve (purple oval), and selects the presumed center of the macula. ROPtool then delineates four posterior pole quadrants by generating a crosshair (green lines) that includes a horizontal line that connects the center of the optic nerve and center of the macula, and a vertical line. If indicated, the tracer then adjusts the quadrant lines to attempt to include at least two major vessels per quadrant. The tracer then selects up to two vessels per quadrant for ROPtool to trace and analyze (red and purple tracings). Example of eyes diagnosed with neither plus nor pre-plus disease (A), pre-plus (B), and plus disease (C) on clinical examination.

For human-expert ranking, 3 fellowship-trained pediatric ophthalmologists who were masked to ROPtool and clinical examination results ranked the vessel maps by increasing vascular tortuosity and/or dilation on a discrete ordinal scale.

R (v3.6.1, R Foundation for Statistical Computing, Vienna, Austria) was used for all statistical analyses. Summary rankings were calculated for ROPtool and human-expert rankings by taking the mean of individual tracer and human-expert rankings, respectively. Inter-tracer and inter-expert ranking agreement was calculated using intraclass correlation coefficient (ICC), where ICC of <0.50 was considered poor; 0.50 ≤ ICC < 0.75, moderate; 0.75 ≤ ICC < 0.90, good; and ICC≥0.90, excellent. Correlation between the mean ROPtool and human-expert rankings were evaluated using Pearson correlation coefficients. ROPtool and human-expert rankings were also compared to the reference standard (ie, clinical examination using binocular indirect ophthalmoscopy).

Results

OCT images from 21 infants (42 eyes) were included. Median gestational age was 26.1 weeks (IQR, 24.7–27.1), birth weight was 790 g (IQR, 620–1090), and postmenstrual age at imaging was 35.3 weeks (IQR, 33.7–36.4). On clinical examination, 1 eye had plus disease, 6 had pre-plus, and 35 had neither.

ROPtool inter-tracer agreement was good for TI (ICC = 0.87) and SAI (ICC = 0.83) rankings, but poor for DI rankings (ICC = 0.36). Human inter-expert agreement was good for tortuosity (ICC = 0.83), dilation (ICC = 0.77), and combined tortuosity/dilation (ICC = 0.88) rankings. The correlation between mean ROPtool and human-expert rankings was strong (r = 0.87) for tortuosity and combined tortuosity/dilation (r = 0.86) rankings and moderate (r = 0.47) for dilation rankings (Figure 2A-C). ROPtool and human-expert rankings correlated well with clinical examination diagnoses (Figure 2D). Ordered by combined ROPtool tortuosity/dilation rankings, the image of plus/pre-plus disease with the lowest rank was 29 of 42. Using this cutoff, among the 14 images with the highest ROPtool tortuosity/dilation rankings, 7 (50%) had plus/pre-plus disease on clinical examination, and of the 7 eyes not clinically diagnosed with plus/pre-plus disease at or before the time of imaging, 6 later developed plus/pre-plus disease and 4 later required treatment.

FIG 2.

Correlation between mean ROPtool and human-expert rankings of optical coherence tomography–generated retinal vessel maps for tortuosity and/or dilation. Correlation was strong (r = 0.87) for tortuosity rankings (A), moderate (r = 0.47) for dilation rankings (B), and strong (r = 0.86) for combined tortuosity/dilation rankings (C). Correlation of ROPtool combined tortuosity/dilation rankings and human-expert rankings with clinical examination diagnosis (D). r, Pearson correlation coefficient.

Discussion

In this proof-of-concept study, we found that bedside OCT-generated retinal vessel maps could be reliably ranked by increasing vascular tortuosity and combined tortuosity/dilation by both a semiautomated computer software program (ie, ROPtool) and human ROP experts, and there was good correlation between their rankings and clinical examination diagnoses.

Reliability between ROPtool tracers was good for tortuosity and combined tortuosity/dilation rankings but poor for dilation rankings. We have previously shown that ROPtool can be used to trace and analyze OCT-generated retinal vessel maps to diagnose plus/pre-plus disease with high reliability and accuracy⁶; however, intertracer agreement was lower for dilation than for tortuosity or combined tortuosity/dilation indices, likely due to ROPtool’s inherent inability to consistently identify vessel margins.⁸

There was good agreement between human ROP experts for tortuosity, dilation, and combined tortuosity/dilation rankings. Human experts have been shown to have high agreement on posterior pole vascular disease severity rankings using fundus photographs; eg, one study found an average correlation coefficient of 0.84 (range, 0.74–0.93) for comparison rankings of 34 wide-field fundus photographs among 8 human graders.⁹ Our study found similar agreement on tortuosity (ICC = 0.83), dilation (ICC = 0.77), and combined tortuosity/dilation (ICC = 0.88) rankings of OCT-generated retinal vessel maps. These findings support the theory that plus and pre-plus disease represent a continuum of vascular disease severity.

Finally, ROPtool and human-expert tortuosity and combined tortuosity/dilation rankings correlated well with each other and with clinical examination diagnoses. A previous pilot study, using narrow-field fundus photographs, also found consistent agreement between ROPtool and human-expert rankings, although this agreement was not quantified.¹⁰ A fully automated computer program, i-ROP Deep Learning, also demonstrated good correlation between computer and human rankings (r = 0.86).⁹

Study limitations included the small sample size and small number of eyes with plus/pre-plus disease, in which the reference standard used (i.e., diagnosis on clinical examination) is subjective. We selected only OCT imaging sessions that captured all four posterior pole quadrants, which was not a primary goal of BabySTEPS and may not always be possible in a real-world scenario. Further studies are needed to show whether OCT is a clinically useful adjunct for ROP screening.